WEB2PY Enterprise Web Framework / 2nd Ed.
Massimo Di Pierro Copyright ©2009 by Massimo Di Pierro. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests to the Copyright owner for permission should be addressed to: Massimo Di Pierro School of Computing DePaul University 243 S Wabash Ave Chicago, IL 60604 (USA) Email: [email protected]
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created ore extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging-in-Publication Data:
WEB2PY: Enterprise Web Framework Printed in the United States of America. to my family
CONTENTS
Preface xv
1 Introduction 1 1.1 Principles 3 1.2 WebFrameworks 4 1.3 Model-View-Controller 5 1.4 Whyweb2py 8 1.5 Security 9 1.6 Inthebox 12 1.7 License 13 1.8 License Commercial Exception 14 1.9 Acknowledgments 15 1.10 AboutthisBook 16 1.11 Elementsof Style 18 vii viii CONTENTS
2 ThePythonLanguage 21 2.1 AboutPython 21 2.2 Starting up 22 2.3 help,dir 23 2.4 Types 24 2.5 AboutIndentation 28 2.6 for...in 28 2.7 while 29 2.8 def...return 29 2.9 if...elif...else 31 2.10 try... except...else...finally 31 2.11 class 33 2.12 Special Attributes, Methods and Operators 34 2.13 File Input/Output 34 2.14 lambda 35 2.15 exec, eval 36 2.16 import 37
3 Overview 41 3.1 Startup 41 3.2 SayHello 45 3.3 Let’sCount 50 3.4 SayMyName 51 3.5 Form self-submission 53 3.6 AnImageBlog 56 3.7 AddingCRUD 69 3.8 AddingAuthentication 70 3.9 A Wiki 71 3.10 More on admin 81 [site] 81 [about] 84 [EDIT] 85 [errors] 87 [mercurial] 91 3.11 More on appadmin 91
4 The Core 93 4.1 CommandLineOptions 93 CONTENTS ix
4.2 URLMapping 96 4.3 Libraries 99 4.4 Applications 103 4.5 API 104 4.6 request 105 4.7 response 107 4.8 session 110 4.9 cache 111 4.10 URL 113 4.11 HTTPand redirect 115 4.12 T and Internationalization 116 4.13 Cookies 117 4.14 init Application 118 4.15 URL Rewrite 118 4.16 RoutesonError 120 4.17 Cron 121 4.18 ImportOther Modules 124 4.19 Execution Environment 124 4.20 Cooperation 126
5 The Views 127 5.1 Basic Syntax 129 for...in 129 while 130 if...elif...else 130 try...except...else...finally 131 def...return 131 5.2 HTMLHelpers 132 XML 133 Built-in Helpers 134 Custom Helpers 142 5.3 BEAUTIFY 143 5.4 PageLayout 143 5.5 Using the Template System to Generate Emails 146 5.6 LayoutBuilder 147
6 The Database Abstraction Layer 149 6.1 Dependencies 149 x CONTENTS
6.2 Connection Strings 151 Connection Pooling 152 6.3 DAL, Table, Field 153 6.4 Migrations 154 insert 158 commit and rollback 159 executesql 160 lastsql 160 drop 160 Indexes 160 Legacy Databases 161 Distributed Transaction 161 6.5 Query,Set,Rows 162 select 162 Serializing Rows in Views 164 orderby, groupby, limitby, distinct 164 Logical Operators 165 count, delete, update 166 Expressions 166 update record 166 6.6 OnetoManyRelation 167 Inner Joins 168 Left Outer Join 168 Grouping and Counting 169 6.7 HowtoseeSQL 169 6.8 Exportingand ImportingData 170 CSV (one table at a time) 170 CSV (all tables at once) 170 CSV and remote Database Synchronization 171 HTML/XML (one table at a time) 173 6.9 ManytoMany 173 6.10 Other Operators 175 like, upper, lower 175 year, month, day, hour, minutes, seconds 175 belongs 176 6.11 Caching Selects 176 6.12 Shortcuts 177 6.13 Self-Reference and Aliases 177 CONTENTS xi
6.14 Table Inheritance 179
7 Forms and Validators 181 7.1 FORM 182 Hidden fields 185 keepvalues 186 onvalidation 186 Forms and redirection 187 Multiple forms per page 188 No self-submission 189 7.2 SQLFORM 189 Insert/Update/Delete SQLFORM 193 SQLFORM in HTML 194 SQLFORM and uploads 195 Storing the original filename 197 Removing the action file 198 Links to referencing records 198 Prepopulating the form 200 SQLFORM without database IO 200 7.3 SQLFORM.factory 201 7.4 Validators 202 Basic Validators 203 Database Validators 210 Custom Validators 211 Validators with Dependencies 212 7.5 Widgets 213 7.6 CRUD 214 Attributes 215 Messages 216 Methods 217 7.7 Customform 218 CSS Conventions 220 Switch off errors 220
8 Access Control 223 8.1 Authentication 225 Email verification 227 Restrictions on registration 228 xii CONTENTS
CAPTCHA and reCAPTCHA 228 Customizing Auth 229 Renaming Auth tables 230 Alternate Login Methods 230 8.2 Authorization 233 Decorators 234 Combining requirements 235 Authorization and CRUD 235 Authorization and Downloads 236 Access control and Basic authentication 237 Settings and Messages 237 8.3 Central Authentication Service 241
9 Services 245 9.1 Renderinga dictionary 246 HTML, XML, and JSON 246 How it works 246 Rendering Rows 247 Custom Formats 248 RSS 248 CSV 250 9.2 RemoteProcedureCalls 251 XMLRPC 253 JSONRPC 253 AMFRPC 257 9.3 LowLevelAPIandOtherRecipes 259 simplejson 259 PyRTF 260 ReportLab and PDF 260 9.4 Services and Authentication 261
10 Ajax Recipes 263 10.1 web2py ajax.html 263 10.2 jQuery Effects 268 Conditional Fields in Forms 271 Confirmation on Delete 272 10.3 The ajax Function 274 Eval target 274 CONTENTS xiii
Auto-completion 275 Form Submission 277 Voting and Rating 278
11 Deployment Recipes 281 11.1 Setup Apacheon Linux 284 11.2 Setupmod wsgi on Linux 285 mod wsgi and SSL 287 11.3 Setupmod proxy on Linux 288 11.4 StartasLinuxDaemon 290 11.5 Setup Apacheand mod wsgi on Windows 291 11.6 Start as Windows Service 293 11.7 Setup Lighttpd 294 11.8 Apache2andmod pythoninasharedhostingenvironment 295 11.9 Setup Cherokee with FastGGI 296 11.10 Setup PostgreSQL 297 11.11 Security Issues 298 11.12 Scalability Issues 299 Sessions in Database 300 Pound, a High Availability Load Balancer 301 Cleanup Sessions 301 Upload Files in Database 302 Collecting Tickets 303 Memcache 304 Sessions in Memcache 305 Removing Applications 305 11.13 Google App Engine 305
12 Other Recipes 309 12.1 Upgrading web2py 309 12.2 FetchingaURL 310 12.3 Geocoding 310 12.4 Pagination 310 12.5 Streaming Virtual Files 311 12.6 httpserver.log and the log file format 312 12.7 SendanSMS 313 12.8 Twitter API 314 12.9 Jython 314 xiv CONTENTS
References 317 Preface
I am guilty! After publicly complaining about the existence of too many Python based web frameworks, after praising the merits of Django, Pylons, TurboGears, CherryPy, and web.py, after having used them professionally and taught them in University level courses, I could not resist and created one more: web2py. Why did I commit such a crime? I did it because I felt trapped by existing choices and tempted by the beautiful features of the Python language. It all started with the need to convince my father to move away from Visual Basic and embrace Python as a development language for the Web. At the same time I was teaching a course on Python and Django at DePaul University. These two experiences made me realize how the beautiful features of those systems were hidden behind a steep learning curve. At the University for example we teach introductory programming using languages like Java and C++ but we do not get into networking issues until later courses. In many Universities students can graduate in Computer Science without ever seeing a Unix Bash Shell or editing an Apache configuration file. And yet these days to be an effective web developer you must know shell scripting, Apache, SQL, HTML, CSS, JavaScript, and Ajax. Knowing how to program in one
xv xvi PREFACE language is not enough to understand the intricacy and subtleties of the APIs exposed by the existing frameworks. Not to mention security. web2py started with the goal to drastically reduce the learning curve, incorporating everything needed into a single tool that is accessible via the web browser, collapsing the API to a minimum (only 12 core objects and functions), delegating all the security issues to the framework, and forcing developers to follow modern software engineering practices. Most of the development work was done in the summer of 2007 while I was on vacation. Since web2py was released many people have contributed by submitting patches to fix bugs and to add features. web2py has evolved steadily since and yet it never broke backward compatibility. In fact, web2py has a top-down design vs the bottom-up design of other frameworks. It is not built by adding layer upon layer. It is built from the user perspective and it has been constantly optimized inside in order to become faster and leaner, while always keeping backward compatibility. I am happy to say that today web2py is one of the fastest web frameworks and also one of the the smallest (the core libraries including the Database Abstraction Layer, the template language, and all the helpers amounts to about 300KB, the entire source code including sample applications and images amounts to less than 2.0MB). Yes, I am guilty, but so are the growing number of users and contributors. Nevertheless, I feel, I am no more guilty than the creators of the other frameworks I have mentioned. Finally, I would like to point out, I have already paid a price for my crime, since I have been condemned to spend my 2008 summer vacation writing this book and my 2009 summer vacations revising it. This second edition describes many features added after the release of the first edition, including CRUD, Access Control, and Services. I hope you, dearreader, understandI have done it for you: to free you from current web programming difficulties, and to allow you to express yourself more and better on the Web. CHAPTER 1
INTRODUCTION
web2py [1] is a free, open-source web framework for agile development of secure database-driven web applications; it is written in Python[2] and programmable in Python. web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web appli- cations. web2py is designed to guide a web developer to follow good software engineering practices, such as using the Model View Controller (MVC) pat- tern. web2py separates the data representation (the model) from the data presentation (the view) and also from the application logic and workflow (the controller). web2py provides libraries to help the developer design, imple- ment, and test each of these three parts separately, and makes them work together. web2py is built for security. This means that it automatically addresses many of the issues that can lead to security vulnerabilities, by following well established practices. For example, it validates all input (to prevent injec- tions), escapes all output (to prevent cross-site scripting), renames uploaded files (to prevent directory traversal attacks), and stores all session information
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 1 Copyright © 2009 2 INTRODUCTION
server side. web2py leaves little choice to application developers in matters related to security. web2py includes a Database AbstractionLayer(DAL)thatwrites SQL [3] dynamically so that the developer does not have to. The DAL knows how to generate SQL transparently for SQLite [4], MySQL [6], PostgreSQL [5], MSSQL [7], FireBird [8], Oracle [9], IBM DB2 [10] and Informix [11]. The DAL can also generate function calls for Google BigTable when running on the Google App Engine (GAE) [12]. Once one or more database tables are defined, web2py also generates a fully functional web-based database administration interface to access the database and the tables. web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer. In fact, web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iPhone, and Unix/Linux), and the development,deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and/or Jython (the Java implementation), versions 2.4, 2.5 and 2.6 although "officially" only support 2.5 else we cannot guarantee backward compatibility for applications. web2py provides a ticketing system. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator. web2py is open source and released under the GPL2.0 license, but web2py developed applications are not subject to any license constraint. As long as applications do not explicitly contain web2py source code, they are not considered "derivative works". web2py also allows the developer to bytecode-compile applications and distribute them as closed source, although they will require web2py to run. The web2py license includes an exception that allows web developers to ship their products with original pre-compiled web2py binaries, without the accompanying source code. Another feature of web2py, is that we, its developers, commit to maintain backward compatibility in future versions. We have done so since the first release of web2py in October, 2007. Newfeatureshavebeenaddedandbugs have been fixed, but if a program worked with web2py 1.0, that program will still work today. Here are some examples of web2py statements that illustrate its power and simplicity. The following code:
1 db.define_table('person', 2 Field('name', 'string'), 3 Field('image', 'upload')) PRINCIPLES 3
creates a database table called "person" with two fields: "name", a string; and "image", something that needs to be uploaded (the actual image). If the table already exists but does not match this definition, it is altered appropriately. Given the table defined above, the following code:
1 form = SQLFORM(db.person) creates an insert form for this table that allows users to upload images. The following statement:
1 if form.accepts(request.vars, session): 2 pass validates a submitted form, renames the uploaded image in a secure way, stores the image in a file, inserts the corresponding record in the database, prevents double submission, and eventually modifies the form itself by adding error messages if the data submitted by the user does not pass validation.
1.1 Principles
Python programming typically follows these basic principles: • Don’t repeat yourself (DRY). • There should be only one way of doing things. • Explicit is better than implicit. web2py fully embraces the first two principles by forcing the developer to use sound software engineering practices that discourage repetition of code. web2py guides the developer through almost all the tasks common in web application development (creating and processing forms, managing sessions, cookies, errors, etc.). web2py differs from other frameworks with regard to the third principle, which sometimes conflicts with the other two. In particular, web2py auto- matically imports its own modules and instantiates its global objects (request, response, session, cache, T) and this is done "under the hood". To some this may appear as magic, but it should not. web2py is trying to avoid the an- noying characteristic of other frameworks that force the developer to import the same modules at the top of every model and controller. web2py, by importing its own modules, saves time and prevents mistakes, thus following the spirit of "don’t repeat yourself" and "there should be only one way of doing things". If the developer wishes to use other Python modules or third-party modules, those modules must be imported explicitly, as in any other Python program. 4 INTRODUCTION
1.2 Web Frameworks
At its most fundamental level, a web application consists of a set of programs (or functions) that are executed when a URL is visited. The output of the program is returned to the visitor and rendered by the browser. The two classic approaches for developing web applications are:
• Generating HTML [13, 14] programmatically and embedding HTML as strings into computer code.
• Embedding pieces of code into HTML pages.
The first modelis the one followed, for example, by early CGI scripts. The second model is followed, for example, by PHP [15] (where the code is in PHP, a C-like language), ASP (where the code is in Visual Basic), and JSP (where the code is in Java). Here we present an example of a PHP program that, when executed, retrieves data from a database and returns an HTML page showing the selected records:
1
Records
echo "$namePhone:$phone
"; 12 $i++; 13 } 14 ?>
The problem with this approach is that code is embedded into HTML, but this very same code also needs to generate additional HTML and to generate SQL statements to query the database, entangling multiple layers of the application and making it difficult to read and maintain. The situation is even worse for Ajax applications, and the complexity grows with the number of pages (files) that make up the application. The functionality of the above example can be expressed in web2py with two lines of Python code:
1 def index(): 2 return HTML(BODY(H1('Records'), db().select(db.contacts.ALL))) MODEL-VIEW-CONTROLLER 5
In this simple example, the HTML page structure is represented program- matically by the HTML, BODY, and H1 objects; the database db1 is queried by the select command; finally, everything is serialized into HTML. This is just one exampleof the power of web2py and its built-in libraries. web2py does even more for the developer by automatically handling cookies, sessions, creation of database tables, database modifications, form validation, SQL injection prevention, cross-site scripting (XSS) prevention, and many other indispensable web application tasks. Web frameworks are typically categorized as one of two types: A "glued" framework is built by assembling (gluing together) several third-party com- ponents. A "full-stack" framework is built by creating components designed specifically to work together and be tightly integrated. web2py is a full-stack framework. Almost all of its components are built from scratch and designed to work together, but they function just as well outside of the complete web2py framework. For example, the Database Abstraction Layer (DAL) or the template language can be used independently of the web2py framework by importing gluon.sql or gluon.template into your own Python applications. gluon is the name of the web2py folder that contains system libraries. Some web2py libraries, such as building and processing forms from database tables, have dependencies on other portions of web2py. web2py can also work with third-party Python libraries, including other template languages and DALs, but they will not be as tightly integrated as the original components.
1.3 Model-View-Controller web2py forces the developer to separate data representation (the model), data presentation (the view) and the application workflow (the controller). Let’s consider again the previous example and see how to build a web2py application around it.
1There is nothing special about the name db; it is just a variable holding your database connection. 6 INTRODUCTION
The typical workflow of a request in web2py is described in the following diagram:
In the diagram: • The Server can be the web2py built-in web server or a third-party server, such as Apache. The Server handles multi-threading. • Main is the main web2py WSGI application. It performs all common tasks and wraps user applications. It deals with cookies, sessions, transactions, url mapping and reverse mapping, dispatching (deciding which function to call based on the URL). It can serve and stream static files if the web server is not doing it already. • The Models, Views and Controller components make up the user appli- cation. There can be multiple applications hosted in the same web2py instance. • The dashed arrows represent communication with the database engine (or engines). The database queries can be written in raw SQL (discour- aged) or by using the web2py Database Abstraction Layer (recom- mended), so that that web2py application code is not dependent on the specific database engine. • The dispatcher maps the requested URL into a function call in the controller. The output of the function can be a string or a dictionary MODEL-VIEW-CONTROLLER 7
of symbols (a hash table). The data in the dictionary is rendered by a view. If the visitor requests an HTML page (the default), the dictionary is rendered into an HTML page. If the visitor requests the same page in XML, web2py tries to find a view that can render the dictionary in XML. The developer can create views to render pages in any of the already supported protocols (HTML, XML, JSON, RSS, CSV,RTF) or additional custom protocols.
• All calls are wrapped into a transaction, and any uncaught exception causes the transaction to roll back. If the request succeeds, the trans- action is committed.
• web2py also handles sessions and session cookies automatically, and when a transaction is committed, the session is also stored.
• It is possible to register recurrent tasks (cron) to run at scheduled times and/or after the completion of certain actions. In this way it is possible to run long and compute-intensive tasks in the background without slowing down navigation. Here is a minimal and complete MVC application consisting of three files:
• "db.py" is the model:
1 db = DAL('sqlite://storage.sqlite') 2 db.define_table('contacts', 3 Field('name'), 4 Field('phone'))
It connects to the database (in this example a SQLite database stored in the storage.sqlite file) and defines a table called contacts. If the table does not exist, web2py creates it and, transparently and in the background, generates SQL code in the appropriate SQL dialect for the specificdatabaseengineused. ThedevelopercanseethegeneratedSQL but does not need to change the code if the database back-end, which defaults to SQLite, is replaced with MySQL, PostgreSQL, MSSQL, FireBird, Oracle, DB2, Informix, or Google Big Tables in the Google App Engine. Once a table is defined and created, web2py also generates a fully functional web-based database administration interface to access the database and the tables. It is called appadmin.
• "default.py" is the controller:
1 def contacts(): 2 return dict(records=db().select(db.contacts.ALL)) 8 INTRODUCTION
In web2py, URLs are mapped to Python modules and function calls. In this case, the controller contains a single function (or "action") called contacts. An action may return a string (the returned website) or a Python dictionary (a set of key:value pairs). If the function returns a dictionary, it is passed to a view with the same name as the con- troller/function, which in turn renders it. In this example, the function contacts performs a database select and returns the resulting records as a value associated with the dictionary key records. • "default/contacts.html" is the view:
1 {{extend 'layout.html'}} 2
Records
3 {{for record in records:}} 4 {{=record.name}}: {{=record.phone}}5 {{pass}}
This view is called automatically by web2py after the associated controller function (action) is executed. The purpose of this view is to render the variables in the returned dictionary records=... into HTML. The view file is written in HTML, but it embeds Python code delimited by the special {{ and }} delimiters. This is quite different from the PHP code example, because the only code embedded into the HTML is "presentation layer" code. The "layout.html" file referenced at the top of the view is provided by web2py and constitutes the basic layout for all web2py applications. The layout file can easily be modified or replaced.
1.4 Why web2py web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations: • Easy for users to learn server-side web development without compro- mising on functionality. For this reason web2py requires no installa- tion, no configuration, has no dependencies2, and exposes most of its functionality via a web interface. • web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even
2except for the source code distribution, which requires Python 2.5 and its standard library modules SECURITY 9
as new functionality has been added, web2py has never broken back- wards compatibility, and it will not break compatibility when additional functionality is added in the future. • web2py proactively addresses the most important security issues that plague many modern web applications, as determined by OWASP[19] below. • web2py is light. Its core libraries, including the Database Abstraction Layer, the template language,andall the helpers amountto 300KB.The entire source code including sample applications and images amounts to 2.0MB. • web2py hasasmallfootprintandisveryfast. ItusestheCherryPy[16] WSGI-compliant3 web server that is 30% faster than Apache with mod proxy and four times faster than the Paste http server. Our tests also indicate that, on an average PC, it serves an average dynamic page without database access in about 10ms. The DAL has very low overhead, typically less than 3%.
1.5 Security
The Open Web Application Security Project[19] (OWASP) is a free and open worldwide community focused on improving the security of application software. OWASP has listed the top ten security issues that put web applications at risk. That list is reproduced here, along with a description of how each issue is addressed by web2py: • "Cross Site Scripting (XSS): XSS flaws occur whenever an application takes user supplied data and sends it to a web browser without first validating or encoding that content. XSS allows attackers to execute scripts in the victim’s browser which can hijack user sessions, deface web sites, possibly introduce worms, etc." web2py, by default, escapes all variables rendered in the view, pre- venting XSS. • "Injection Flaws: Injection flaws, particularly SQL injection, are com- mon in web applications. Injection occurs when user-supplied data is
3The Web Server Gateway Interface [17, 18] (WSGI) is an emerging Python standard for communication between a web server and Python applications. 10 INTRODUCTION
sent to an interpreter as part of a command or query. The attacker’s hostile data tricks the interpreter into executing unintended commands or changing data." web2py includes a Database Abstraction Layer that makes SQL in- jection impossible. Normally, SQL statements are not written by the developer. Instead, SQL is generated dynamically by the DAL, ensur- ing that all inserted data is properly escaped.
• "Malicious File Execution: Code vulnerable to remote file inclusion (RFI) allows attackers to include hostile code and data, resulting in devastating attacks, such as total server compromise." web2py allows only exposed functions to be executed, preventing malicious file execution. Imported functions are never exposed; only actions are exposed. web2py’s web-based administration interface makes it very easy to keep track of what is exposed and what is not.
• "Insecure Direct Object Reference: A direct object reference occurs when a developer exposes a reference to an internal implementation object, such as a file, directory, database record, or key, as a URL or form parameter. Attackers can manipulate those references to access other objects without authorization." web2py does not exposeanyinternal objects; moreover, web2py val- idates all URLs, thus preventing directory traversal attacks. web2py also provides a simple mechanism to create forms that automatically validate all input values.
• "Cross Site Request Forgery (CSRF): A CSRF attack forces a logged- on victim’s browser to send a pre- authenticated request to a vulnerable web application, which then forces the victim’s browser to perform a hostile action to the benefit of the attacker. CSRF can be as powerful as the web application that it attacks." web2py stores all session information server side, and storing only the session id in a browser-side cookie; moreover, web2py prevents double submission of forms by assigning a one-time random token to each form.
• "Information Leakage and Improper Error Handling: Applications can unintentionally leak information about their configuration, internal workings, or violate privacy through a variety of application problems. Attackers use this weakness to steal sensitive data, or conduct more serious attacks." web2py includes a ticketing system. No error can result in code being exposed to the users. All errors are logged and a ticket is issued to the SECURITY 11
user that allows error tracking. Errors and source code are accessible only to the administrator.
• "Broken Authentication and Session Management: Account creden- tials and session tokens are often not properly protected. Attackers compromise passwords, keys, or authentication tokens to assume other users’ identities." web2py provides a built-in mechanism for administrator authentica- tion, and it manages sessions independently for each application. The administrative interface also forces the use of secure session cook- ies when the client is not "localhost". For applications, it includes a powerful Role Based Access Control API.
• "Insecure Cryptographic Storage: Web applications rarely use crypto- graphic functions properly to protect data and credentials. Attackers use weakly protected data to conduct identity theft and other crimes, such as credit card fraud." web2py uses the MD5 or the HMAC+SHA-512 hash algorithms to protect stored passwords. Other algorithms are also available.
• "Insecure Communications: Applications frequently fail to encrypt network traffic when it is necessary to protect sensitive communica- tions." web2py includes the SSL-enabled [20] CherryPy WSGI server, but it canalsouseApacheorLighttpd andmod ssl to provide SSL encryption of communications.
• "Failure to Restrict URL Access: Frequently an application only pro- tects sensitive functionality by preventing the display of links or URLs to unauthorized users. Attackers can use this weakness to access and perform unauthorized operations by accessing those URLs directly." web2py mapsURLrequeststoPythonmodulesandfunctions. web2py provides a mechanism for declaring which functions are public and which require authentication and authorization. The included Role Based Access Control API allow developers to restcrict access to any function based on login, group membership or group based permis- sions. The permissions are very granular and can be combined with CRUD to allow, for example, to give access to specific tables and/or records. 12 INTRODUCTION
1.6 In the box
You can download web2py from the official web site:
http://www.web2py.com web2py is composed of the following components:
• libraries: provide core functionality of web2py and are accessible programmatically.
• web server: the CherryPy WSGI web server.
• the admin application: used to create, design, and manage other web2py applications. admin provide a complete web-based Inte- grated Development Environment (IDE) for building web2py appli- cations. It also includes other functionality, such as web-based testing and a web-based shell.
• the examples application: contains documentation and interactive ex- amples. examples is a clone of the official web2py web site, and includes epydoc and Sphinx documentation.
• the welcome application: the basic scaffolding template for any other application. By default it includes a pure CSS cascading menu and user authentication (discussed in Chapter 8).
web2py is distributed in source code and binary form for Microsoft Windows and for Mac OS X. The source code distribution can be used in any platform where Python or Jython run, and includes the above-mentioned components. To run the source code, you need Python 2.5 pre-installed on the system. You also need one of the supported database engines installed. For testing and light-demand applications, you can use the SQLite database, included with Python 2.5. The binary versions of web2py (for Windows and Mac OS X) include a Python 2.5 interpreter and the SQLite database. Technically, these two are not components of web2py. Including them in the binary distributions enables you to run web2py out of the box. The following image depicts the overall web2py structure: LICENSE 13
1.7 License web2py is licensed under the GPL version 2 License. The full text of the license if available in ref. [30]. The license includes but it is not limited to the following articles: 1. You may copy and distribute verbatim copies of the Program’s source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keepintact all the notices that refer to this Licenseand to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. [...] 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. [...] 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER 14 INTRODUCTION
PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABIL- ITY AND FITNESS FOR A PARTICULARPURPOSE.THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU AS- SUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR COR- RECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR IN- ABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. • web2py includes some third-party code (for example the Python in- terpreter, the CherryPy web server, and some JavaScript libraries). Their respective authors and licenses are acknowledged in the official website [1] and in the code itself. • Applications developed with web2py, as long as they do not include web2py source code, are not considered derivativeworks. This means they are not bound by the GPLv2 license, and you can distribute the applications you developed under any license you choose, including a closed-source and/or commercial license.
1.8 License Commercial Exception
The web2py license also includes a commercial exception: You may distribute an application you developed with web2py together with an unmodified official binary distribution of web2py, as downloaded from the official website[1], as long as you make it clear in the license of your application which files belong to the application and which files belong to web2py. ACKNOWLEDGMENTS 15
1.9 Acknowledgments web2py was originally developed by and copyrighted by Massimo Di Pierro. The first version (1.0) was released in October, 2007. Since then it has been adopted by many users, some of whom have also contributed bug reports, testing, debugging, patches, and proofreading of this book. Some of the major contributors are, in alphabetical order by first name: Alexandre Andrade, Alexey Nezhdanov(GAE and database performance), Alvaro Justen (dynamical translations), Andre Berthiaume, Andre Bossard, Attila Csipa (cron job), Bill Ferrett (modular DAL design), Boris Manojlovic (Ajax edit), Carsten Haese (Informix), Chris Baron, Christopher Smiga (In- formix), Clifford John Lazell (tester and JS), David H. Lee (OpenID), Denes Lengyel (validators, DB2 support), Douglas Soares de Andrade (2.4 and 2.6 compliance, docstrings), Felipe Barousse, Fran Boon (authorization and au- thentication), Francisco Gama (bug fixing), Fred Yankowski (XHTML com- pliance), Gabriele Carrettoni, Graham Dumpleton, Gregor Jovanovich, Hans Christian v. Stockhausen (OpenID), Hans Donner (GAE support, Google login, widgets, Sphinx documentation), Ivan Valev, Joe Barnhart, Jonathan Benn (validators and tests), Jonathan Lundell, Jose Jachuf (Firebird sup- port), Kacper Krupa, Kyle Smith (JavaScript), Limodou (winservice), Lucas Geiger,Marcel Leuthi (Oracle support), Mark Larsen (taskbar widget), Mark Moore (databases and daemon scripts), Markus Gritsch (bug fixing), Martin Hufsky (expressions in DAL), Mateusz Banach (stickers, validators, content- type), Michael Willis (shell), Milan Andric, Minor Gordon, Nathan Freeze (admin design, validators), Niall Sweeny (MSSQL support), Niccolo Polo (epydoc), Nicolas Bruxer (memcache support), Ondrej Such (MSSQL sup- port), Pai (internationalization), Phyo Arkar Lwin (web hosting and Jython tester), Ricardo Cardenes, Richard Gordon, Richard Baron Penman, Robin Bhattacharyya (Google App Engine support), Roman Goldmann, Ruijun Luo (windows binary), Scott Santarromana, Sergey Podlesnyi (Oracle and migra- tions tester), Shane McChesney, Sharriff Aina (tester and PyAMF integra- tion), Sterling Hankins (tester), Stuart Rackham (MSSQL support), Telman Yusupov (Oracle support), Tim Farrell, Tim Michelsen (Sphinx documen- tation), Timothy Farrell (Python 2.6 compliance, windows support), Tito Garrido, Yair Eshel (internationalizaiton), Yarko Tymciurak (design, Sphinx documentation), Ygao, Younghyun Jo (internationalization), Zoom Quiet I am sure I forgot somebody, so I apologize. I particularly thank Alvaro, Denes, Felipe, Graham, Jonathan, Hans, Kyle, Mark, Richard, Robin, Roman, Scott, Shane, Sharriff, Sterling, Stuart and Yarko for proofreading various chapters of this book. Their contribution was invaluable. If you find any errors in this book, they are exclusively my fault, 16 INTRODUCTION probably introduced by a last-minute edit. I also thank Ryan Steffen of Wiley Custom Learning Solutions for help with publishing the first edition of this book. web2py contains code from the following authors, whom I would like to thank: Guido van Rossum for Python [2], Peter Hunt, Richard Gordon, Robert Brewer for the CherryPy [21] web server, Christopher Dolivet for EditArea [22], Brian Kirchoff for nicEdit [23], Bob Ippolito for simplejson [24], Simon Cusack and Grant Edwards for pyRTF [25], Dalke Scientific Software for pyRSS2Gen [26], Mark Pilgrim for feedparser [27], Trent Mick for mark- down2 [28], Allan Saddi for fcgi.py, Evan Martin for the Python memcache module [29], John Resig for jQuery [31]. The logo used on the cover of this book was designed by Peter Kirchner at Young Designers. I thank Helmut Epp (provost of DePaul University), David Miller (Dean of the College of Computing and Digital Media of DePaul University), and Estia Eichten (Member of MetaCryption LLC), for their continuous trust and support. Finally, Iwishtothankmywife,Claudia,andmyson,Marco,forputtingup with me during the many hours I have spent developing web2py, exchanging emails with users and collaborators, and writing this book. This book is dedicated to them.
1.10 About this Book
This book includes the following chapters, besides this introduction:
• Chapter 2 is a minimalist introduction to Python. It assumes knowl- edge of both procedural and object-oriented programming concepts such as loops, conditions, function calls and classes, and covers basic Python syntax. It also covers examples of Python modules that are used throughout the book. If you already know Python, you may skip Chapter 2.
• Chapter 3 shows how to start web2py, discusses the administrative interface, and guides the reader through various examples of increasing complexity: an application that returns a string, a counter application, an image blog, and a full blown wiki application that allows image uploads and comments, provides authentication, authorization, web services and an RSS feed. While reading this chapter, you may need ABOUT THIS BOOK 17
to refer to Chapter 2 for general Python syntax and to the following chapters for a more detailed reference about the functions that are used. • Chapter 4 covers more systematically the core structure and libraries: URL mapping, request, response, sessions, cacheint, CRON, interna- tionalization and general workflow. • Chapter 5 is a reference for the template language used to build views. It shows how to embed Python code into HTML, and demonstrates the use of helpers (objects that can generate HTML). • Chapter 6 covers the Database Abstraction Layer, or DAL. The syntax of the DAL is presented through a series of examples. • Chapter 7 covers forms, form validation and form processing. FORM is the low level helper for form building. SQLFORM is the high level form builder. In Chapter 7 we also discuss the new Create/Read/Up- date/Delete (CRUD) API. • Chapter 8 covers authentication, authorization and the extensible Role- Based Access Control mechanism available in web2py. Mail config- uration and CAPTCHA are also discussed here, since they are used by authentication. • Chapter 9 is about creating web services in web2py. We provide examples of integration with the Google Web Toolkit via Pyjamas, and with Adobe Flash via PyAMF. • Chapter 10 is about web2py and jQuery recipes. web2py is designed mainly for server-side programming, but it includes jQuery, since we have found it to be the best open-source JavaScript library available for effects and Ajax. In this chapter, we discuss how to effectively use jQuery with web2py. • Chapter 11 is about production deployment of web2py applications. We mainly address three possible production scenarios: on a Linux web server or a set of servers (which we consider the main deployment alternative), running as a service on a Microsoft Windows environment, and deployment on the Google Applications Engine (GAE). In this chapter, we also discuss security and scalability issues. • Chapter 12 contains a variety of other recipes to solve specific tasks, inlcuding upgrades, gecoding, pagination, Twitter API, and more. This book only covers basic web2py functionalities and the API that ships with web2py. This book does not cover web2py appliances, for 18 INTRODUCTION example KPAX, the web2py Content Management System. The appliance for Central Authentication Service is briefly discussed in Chapter 8. You can download web2pyappliances fromthe corresponding website [33]. You can find additional topics discussed on AlterEgo [34], the interactive web2py FAQ.
1.11 Elements of Style
Ref. [35] contains good style practices when programming with Python. You will find that web2py does not always follow these rules. This is not because of omissions or negligence; it is our belief that the users of web2py should follow these rules and we encourage it. We chose not to follow some of those rules when defining web2py helper objects in order to minimize the probability of name conflict with objects defined by the user. For example, the class that represents a
• HTML helpers and validators are all upper case for the reasons dis- cussed above (for example DIV, A, FORM, URL).
• The translator object T is uppercasedespitethe fact thatit is an instance of a class and not a class itself. Logically the translator object performs an action similar to the HTML helpers — it affects rendering part of the presentation. Also, T needs to be easy to locate in the code and has to have a short name.
• DAL classes follow the Python style guide (first letter capitalized), sometimes with the addition of a clarifying DAL prefix (for example Table, Field, DALQuery, etc.). ELEMENTS OF STYLE 19
In all other cases we believe we have followed, as much as possible, the Python Style Guide (PEP8). For example all instance objects are lower-case (request, response, session, cache), and all internal classes are capitalized. In all the examples of this book, web2py keywords are shown in bold, while strings and comments are shown in italic.
CHAPTER 2
THE PYTHON LANGUAGE
2.1 About Python
Python is a general-purpose and very high-level programming language. Its design philosophy emphasizes programmer productivity and code readability. It has a minimalist core syntax with very few basic commands and simple semantics, but it also has a large and comprehensive standard library, includ- ing an Application Programming Interface (API) to many of the underlying Operating System (OS) functions. The Python code, while minimalist, de- fines objects such as linked lists (list), tuples (tuple), hash tables (dict), and arbitrarily long integers (long). Python supports multiple programming paradigms. These are object- oriented (class), imperative (def), and functional (lambda) programming. Python has a dynamic type system and automatic memory management using reference counting (similar to Perl, Ruby, and Scheme). Python was first released by Guido van Rossum in 1991. The languagehas an open, community-based development model managed by the non-profit Python Software Foundation. There are many interpreters and compilers that
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 21 Copyright © 2009 22 THE PYTHON LANGUAGE
implement the Python language, including one in Java (Jython) but, in this brief review, we refer to the reference C implementation created by Guido. You can find many tutorials, the official documentation and library refer- ences of the language on the official Python website [2] For additional Python references, we can recommend the books in ref. [36] and ref. [37]. You may skip this chapter if you are already familiar with the Python language.
2.2 Starting up
The binary distributions of web2py for Microsoft Windows or Apple OS X come packaged with the Python interpreter built into the distribution file itself. You can start it on Windows with the following command (type at the DOS prompt):
1 web2py.exe -S welcome On Apple OS X, enter the following command type in a Terminal window (assuming you’re in the same folder as web2py.app):
1 ./web2py.app/Contents/MacOS/web2py -S welcome On a Linux or other Unix box, chances are that you have Python already installed. If so, at a shell prompt type:
1 python web2py.py -S welcome If you do not have Python 2.5 already installed, you will have to download and install it before running web2py. The -S welcome command line option instructs web2py to run the inter- active shell as if the commands were executed in a controller for the welcome application, the web2py scaffolding application. This exposes almost all web2py classes, objects and functions to you. This is the only difference between the web2py interactive command line and the normal Python com- mand line. The admin interface also provides a web-based shell for each application. You can access the one for the "welcome" application at.
1 http://127.0.0.1:8000/admin/shell/index/welcome You can try all the examples in this chapter using the normal shell or the web-based shell. HELP, DIR 23
2.3 help, dir
The Python language provides two commands to obtain documentation about objects defined in the current scope, both builtins and user defined. We can ask for help about an object, for example “1”:
1 >>> help(1) 2 Help on int object: 3 4 class int(object) 5 | int(x[, base]) -> integer 6 | 7 | Convert a string or number to an integer, if possible. A floating point 8 | argument will be truncated towards zero (this does not include a string 9 | representation of a floating point number!) When converting a string, use 10 | the optional base. It is an error to supply a base when converting a 11 | non-string. If the argument is outside the integer range a long object 12 | will be returned instead. 13 | 14 | Methods defined here: 15 | 16 | __abs__(...) 17 | x.__abs__() <==> abs(x) 18 ...
and, since "1" is an integer, we get a description about the int class and all its methods. Here the output has been truncated because it is very long and detailed. Similarly, we can obtain a list of methods of the object "1" with the command dir:
1 >>> dir(1) 2 ['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__ ', '__delattr__', '__div__', '__divmod__', '__doc__', '__float__' , '__floordiv__', '__getattribute__', '__getnewargs__', '__hash__ ', '__hex__', '__index__', '__init__', '__int__', '__invert__', ' __long__', '__lshift__', '__mod__', '__mul__', '__neg__', ' __new__', '__nonzero__', '__oct__', '__or__', '__pos__', '__pow__ ', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__ ', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', ' __rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', ' __rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__ ', '__str__', '__sub__', '__truediv__', '__xor__'] 24 THE PYTHON LANGUAGE
2.4 Types
Python is a dynamically typed language, meaning that variables do not have a type and therefore do not have to be declared. Values, on the other hand, do have a type. You can query a variable for the type of value it contains:
1 >>> a = 3 2 >>> print type(a) 3
str Python supports the use of two different types of strings: ASCII strings and Unicode strings. ASCII strings are delimited by ’...’, "..." or by ”’..”’ or """...""". Triple quotes delimit multiline strings. Unicode strings start with a u followed by the string containing Unicode characters. A Unicode string can be converted into an ASCII string by choosing an encoding for example:
1 >>> a = 'this is an ASCII string' 2 >>> b = u'This is a Unicode string' 3 >>> a = b.encode('utf8')
After executing these three commands, the resulting a is an ASCII string storing UTF8 encoded characters. By design, web2py uses UTF8 encoded strings internally. It is also possible to write variables into strings in various ways:
1 >>> print 'number is ' + str(3) 2 number is 3 3 >>> print 'number is %s' % (3) 4 number is 3 5 >>> print 'number is %(number)s' % dict(number=3) 6 number is 3 Thelastnotationis moreexplicitandlesserrorprone,andisto be preferred. Many Python objects, for example numbers, can be serialized into strings using str or repr. These two commands are very similar but produce slightly different output. For example:
1 >>> for i in [3, 'hello']: 2 print str(i), repr(i) 3 3 3 4 hello 'hello' TYPES 25
For user-defined classes, str and repr can be defined/redefined using the special operators str and repr . These are briefly described later on; for more, refer to the official Python documentation [38]. repr always has a default value. Another important characteristic of a Python string is that, like a list, it is an iterable object.
1 >>> for i in 'hello': 2 print i 3 h 4 e 5 l 6 l 7 o
list The main methods of a Python list are append, insert, and delete:
1 >>> a = [1, 2, 3] 2 >>> print type(a) 3
1 >>> print a[:3] 2 [2, 7, 3] 3 >>> print a[1:] 4 [7, 3, 8] 5 >>> print a[-2:] 6 [3, 8] and concatenated:
1 >>> a = [2, 3] 2 >>> b = [5, 6] 3 >>> print a + b 4 [2, 3, 5, 6] A list is iterable; you can loop over it:
1 >>> a = [1, 2, 3] 2 >>> for i in a: 3 print i 4 1 5 2 6 3 26 THE PYTHON LANGUAGE
The elements of a list do not have to be of the same type; they can be any type of Python object.
tuple A tuple is like a list, but its size and elements are immutable, while in a list they are mutable. If a tuple element is an object, the object attributes are mutable. A tuple is delimited by round brackets.
1 >>> a = (1, 2, 3) So while this works for a list:
1 >>> a = [1, 2, 3] 2 >>> a[1] = 5 3 >>> print a 4 [1, 5, 3] the element assignment does not work for a tuple:
1 >>> a = (1, 2, 3) 2 >>> print a[1] 3 2 4 >>> a[1] = 5 5 Traceback (most recent call last): 6 File "
1 >>> a = (1) 2 >>> print type(a) 3
1 >>> a = 2, 3, 'hello' 2 >>> x, y, z = a 3 >>> print x 4 2 5 >>> print z 6 hello
dict
APython dictionary is a hash table that maps a key object to a value object. For example: TYPES 27
1 >>> a = {'k':'v', 'k2':3} 2 >>> a['k'] 3 v 4 >>> a['k2'] 5 3 6 >>> a.has_key('k') 7 True 8 >>> a.has_key('v') 9 False Keys can be of any hashable type (int, string, or any object whose class implements the hash method). Values can be of any type. Different keys and values in the same dictionary do not have to be of the same type. If the keys are alphanumeric characters, a dictionary can also be declared with the alternative syntax:
1 >>> a = dict(k='v', h2=3) 2 >>> a['k'] 3 v 4 >>> print a 5 {'k':'v', 'h2':3}
Useful methods are has key, keys, values and items:
1 >>> a = dict(k='v', k2='3) 2 >>> print a.keys() ,'3 ['k' k2'] 4 >>> print a.values() 5 ['v',3] 6 >>> print a.items() ,'7 [('k' v'), ('k2', 3)]
The items method produces a list of tuples, each containing a key and its associated value. Dictionary elements and list elements can be deleted with the command del:
1 >>> a = [1, 2, 3] 2 >>> del a[1] 3 >>> print a 4 [1, 3] 5 >>> a = dict(k='v', h2=3) 6 >>> del a['h2'] 7 >>> print a 8 {'k':'v'}
Internally, Python uses the hash operator to convert objects into integers, and uses that integer to determine where to store the value.
1 >>> hash("hello world") 2 -1500746465 28 THE PYTHON LANGUAGE
2.5 About Indentation
Python uses indentation to delimit blocks of code. A block starts with a line ending in colon, and continues for all lines that have a similar or higher indentation as the next line. For example:
1 >>> i = 0 2 >>> while i < 3: 3 >>> print i 4 >>> i = i + 1 5 >>> 6 0 7 1 8 2 It is common to use 4 spaces for each level of indentation. It is a good policy not to mix tabs with spaces, or you may run into trouble.
2.6 for...in
In Python, you can loop over iterable objects:
1 >>> a = [0, 1, 'hello', 'python'] 2 >>> for i in a: 3 print i 4 0 5 1 6 hello 7 python
One common shortcut is xrange, which generates an iterable range without storing the entire list of elements.
1 >>> for i in xrange(0, 4): 2 print i 3 0 4 1 5 2 6 4 This is equivalent to the C/C++/C#/Java syntax:
1 for(int i=0; i<4; i=i+1) { print(i); }
Another useful command is enumerate, which counts while looping:
1 >>> a = [0, 1, 'hello', 'python'] 2 >>> for i, j in enumerate(a): 3 print i, j 4 0 0 5 1 1 6 2 hello 7 3 python WHILE 29
There is also a keyword range(a, b, c) that returns a list of integers starting with the value a, incrementing by c, and ending with the last value smaller than b, a defaults to 0 and c defaults to 1. xrange is similar but does not actually generate the list, only an iterator over the list; thus it is better for looping. You can jump out of a loop using break
1 >>> for i in [1, 2, 3]: 2 print i 3 break 4 1 You can jump to the next loop iteration without executing the entire code block with continue
1 >>> for i in [1, 2, 3]: 2 print i 3 continue 4 print 'test' 5 1 6 2 7 3
2.7 while
The while loop in Python works much as it does in many other programming languages, by looping an indefinite number of times and testing a condition before each iteration. If the condition is False, the loop ends.
1 >>> i = 0 2 >>> while i < 10: 3 i = i + 1 4 >>> print i 5 10
There is no loop...until construct in Python.
2.8 def...return
Here is a typical Python function:
1 >>> def f(a, b=2): 2 return a + b 3 >>> print f(4) 4 6 30 THE PYTHON LANGUAGE
There is no need (or way) to specify types of the arguments or the return type(s). Functionarguments canhave defaultvaluesandcanreturnmultiple objects:
1 >>> def f(a, b=2): 2 return a + b, a - b 3 >>> x, y = f(5) 4 >>> print x 5 7 6 >>> print y 7 3
Function arguments can be passed explicitly by name:
1 >>> def f(a, b=2): 2 return a + b, a - b 3 >>> x, y = f(b=5, a=2) 4 >>> print x 5 7 6 >>> print y 7 -3
Functions can take a variable number of arguments:
1 >>> def f(*a, **b): 2 return a, b 3 >>> x, y = f(3, 'hello', c=4, test='world') 4 >>> print x 5 (3, 'hello') 6 >>> print y 7 {'c':4, 'test':'world'}
Here arguments not passed by name (3, ’hello’) are stored in list a, and arguments passed by name (c and test) are stored in the dictionary b. In the opposite case, a list or tuple can be passed to a function that requires individual positional arguments by unpacking them:
1 >>> def f(a, b): 2 return a + b 3 >>> c = (1, 2) 4 >>> print f(*c) 5 3
and a dictionary can be unpacked to deliver keyword arguments:
1 >>> def f(a, b): 2 return a + b 3 >>> c = {'a':1, 'b':2} 4 >>> print f(**c) 5 3 IF...ELIF...ELSE 31
2.9 if...elif...else
The use of conditionals in Python is intuitive:
1 >>> for i in range(3): 2 >>> if i == 0: 3 >>> print 'zero' 4 >>> elifi==1: 5 >>> print 'one' 6 >>> else: 7 >>> print 'other' 8 zero 9 one 10 other
"elif" means "else if". Both elif and else clauses are optional. There can be more than one elif but only one else statement. Complex conditions can be created using the not, and and or operators.
1 >>> for i in range(3): 2 >>> ifi==0or(i==1andi+1==2): 3 >>> print '0 or 1'
2.10 try... except...else...finally
Python can throw - pardon, raise - Exceptions:
1 >>> try: 2 >>> a = 1 / 0 3 >>> except Exception, e 4 >>> print 'error', e, 'occurred' 5 >>> else: 6 >>> print 'no problem here' 7 >>> finally: 8 >>> print 'done' 9 error 3 occurred 10 done
If the exception is raised, it is caught by the except clause, which is executed, while the else clause is not. If no exception is raised, the except clause is not executed, but the else one is. The finally clause is always executed. There can be multiple except clauses for different possible exceptions:
1 >>> try: 2 >>> raise SyntaxError 3 >>> except ValueError: 4 >>> print 'value error' 5 >>> except SyntaxError: 6 >>> print 'syntax error' 7 syntax error 32 THE PYTHON LANGUAGE
The else and finally clauses are optional. Here is a list of built-in Python exceptions + HTTP (defined by web2py)
1 BaseException 2 +-- HTTP (defined by web2py) 3 +-- SystemExit 4 +-- KeyboardInterrupt 5 +-- Exception 6 +-- GeneratorExit 7 +-- StopIteration 8 +-- StandardError 9 | +-- ArithmeticError 10 | | +-- FloatingPointError 11 | | +-- OverflowError 12 | | +-- ZeroDivisionError 13 | +-- AssertionError 14 | +-- AttributeError 15 | +-- EnvironmentError 16 | | +--IOError 17 | | +--OSError 18 | | +-- WindowsError (Windows) 19 | | +-- VMSError (VMS) 20 | +-- EOFError 21 | +-- ImportError 22 | +-- LookupError 23 | | +-- IndexError 24 | | +--KeyError 25 | +-- MemoryError 26 | +-- NameError 27 | | +-- UnboundLocalError 28 | +-- ReferenceError 29 | +-- RuntimeError 30 | | +-- NotImplementedError 31 | +-- SyntaxError 32 | | +-- IndentationError 33 | | +-- TabError 34 | +-- SystemError 35 | +-- TypeError 36 | +-- ValueError 37 | | +-- UnicodeError 38 | | +-- UnicodeDecodeError 39 | | +-- UnicodeEncodeError 40 | | +-- UnicodeTranslateError 41 +-- Warning 42 +-- DeprecationWarning 43 +-- PendingDeprecationWarning 44 +-- RuntimeWarning 45 +-- SyntaxWarning 46 +-- UserWarning 47 +-- FutureWarning 48 +-- ImportWarning 49 +-- UnicodeWarning
For a detailed description of each of them, refer to the official Python documentation. CLASS 33
web2py exposes only one new exception, called HTTP. When raised, it causes the program to return an HTTP error page (for more on this refer to Chapter 4). Any object can be raised as an exception, but it is good practice to raise objects that extend one of the built-in exceptions.
2.11 class
Because Python is dynamically typed, Python classes and objects may seem odd. In fact, you do not need to define the member variables (attributes) when declaring a class, and different instances of the same class can have different attributes. Attributes are generally associated with the instance, not the class (except when declared as "class attributes", which is the same as "static member variables" in C++/Java). Here is an example:
1 >>> class MyClass(object): pass 2 >>> myinstance = MyClass() 3 >>> myinstance.myvariable = 3 4 >>> print myinstance.myvariable 5 3
Notice that pass is a do-nothing command. In this case it is used to define a class MyClass that contains nothing. MyClass() calls the constructor of the class (in this case the default constructor) and returns an object, an instance of the class. The (object) in the class definition indicates that our class extends the built-in object class. This is not required, but it is good practice. Here is a more complex class:
1 >>> class MyClass(object): 2 >>> z = 2 3 >>> def __init__(self, a, b): 4 >>> self.x= a, self.y =b 5 >>> def add(self): 6 >>> return self.x + self.y + self.z 7 >>> myinstance = MyClass(3, 4) 8 >>> print myinstance.add() 9 9 Functions declared inside the class are methods. Some methods have special reserved names. For example, init is the constructor. All variables are local variables of the method except variables declared outside methods. For example, z is a class variable, equivalent to a C++ static member variable that holds the same value for all instances of the class. Notice that init takes 3 arguments and add takes one, and yet we call them with 2 and 0 arguments respectively. The first argument represents, 34 THE PYTHON LANGUAGE
by convention, the local name used inside the method to refer to the current object. Here we use self to refer to the current object, but we could have used any other name. self plays the same role as *this in C++ or this in Java, but self is not a reserved keyword. This syntax is necessary to avoid ambiguity when declaring nested classes, such as a class that is local to a method inside another class.
2.12 Special Attributes, Methods and Operators
Class attributes, methods, and operators starting with a double underscore are usually intended to be private, although this is a convention that is not enforced by the interpreter. Some of them are reserved keywords and have a special meaning. Here, as an example, are three of them:
• len
• getitem
• setitem They can be used, for example, to create a container object that acts like a list:
1 >>> class MyList(object) 2 >>> def __init__(self, *a): self.a = a 3 >>> def __len__(self): return len(self.a) 4 >>> def __getitem__(self, i): return self.a[i] 5 >>> def __setitem__(self, i, j): self.a[i] = j 6 >>> b = MyList(3, 4, 5) 7 >>> print b[1] 8 4 9 >>> a[1] = 7 10 >>> print b.a 11 [3, 7, 5]
Other special operators include getattr and setattr , which define the get and set attributes for the class, and sum and sub , which overload arithmetic operators. For the use of these operators we refer the reader to more advanced books on this topic. We have already mentioned the special operators str and repr .
2.13 File Input/Output
In Python you can open and write in a file with: LAMBDA 35
1 >>> file = open('myfile.txt', 'w') 2 >>> file.write('hello world') Similarly, you can read back from the file with:
1 >>> file = open('myfile.txt', 'r') 2 >>> print file.read() 3 hello world Alternatively, you can read in binary mode with "rb", write in binary mode with "wb", and open the file in append mode "a", using standard C notation. The read command takes an optional argument, which is the number of bytes. You can also jump to any location in a file using seek. You can read back from the file with read
1 >>> print file.seek(6) 2 >>> print file.read() 3 world and you can close the file with:
1 >>> file.close() although often this is not necessary, because a file is closed automatically when the variable that refers to it goes out of scope. When using web2py, you do not know where the current direc- tory is, because it depends on how web2py is configured. The variable request.folder contains the path to the currentapplica- tion. Pathscanbe concatenatedwith the command os.path.join, discussed below.
2.14 lambda
There are cases when you may need to dynamically generate an unnamed function. This can be done with the lambda keyword:
1 >>> a = lambda b: b + 2 2 >>> print a(3) 3 5
The expression "lambda [a]:[b]" literally reads as "a function with arguments [a] that returns [b]". Even if the function is unnamed, it can be stored into a variable, and thus it acquires a name. Technically this is different than using def, because it is the variable referring to the function that has a name, not the function itself. Who needs lambdas? Actually they are very useful because they allow to refactor a function into another function by setting default arguments, without defining an actual new function but a temporary one. For example: 36 THE PYTHON LANGUAGE
1 >>> def f(a, b): return a + b 2 >>> g = lambda a: f(a, 3) 3 >>> g(2) 4 5 Here is a more complex and more compelling application. Suppose you have a function that checks whether its argument is prime:
1 def isprime(number): 2 for p in range(2, number): 3 if number % p: 4 return False 5 return True This function is obviously time consuming. Supposeyou have a caching function cache.ram that takes three arguments: a key, a function and a number of seconds.
1 value = cache.ram('key', f, 60)
The first time it is called, it calls the function f(), stores the output in a dictionary in memory (let’s say "d"), and returns it so that value is:
1 value = d['key']=f() The second time it is called, if the key is in the dictionary and not older than the number of seconds specified (60), it returns the corresponding value without performing the function call.
1 value = d['key'] How would you cache the output of the function isprime for any input? Here is how:
1 >>> number = 7 2 >>> print cache.ram(str(number), lambda: isprime(number), seconds) 3 True 4 >>> print cache.ram(str(number), lambda: isprime(number), seconds) 5 True
The output is always the same, but the first time cache.ram is called, isprime is called; the second time it is not.
The existence of lambda allows refactoring an existing function in terms of a different set of arguments. cache.ram and cache.disk are web2py caching functions.
2.15 exec, eval
Unlike Java, Python is a truly interpreted language. This means it has the ability to execute Python statements stored in strings. For example: IMPORT 37
1 >>> a = "print 'hello world'" 2 >>> exec(a) 3 'hello world'
What just happened? The function exec tells the interpreter to call itself and execute the content of the string passed as argument. It is also possible to execute the content of a string within a context defined by the symbols in a dictionary:
1 >>> a = "print b" 2 >>> c = dict(b=3) 3 >>> exec(a, {}, c) 4 3
Here the interpreter, when executing the string a, sees the symbols defined in c (b in the example), but does not see c or a themselves. This is different than a restricted environment, since exec does not limit what the inner code can do; it just defines the set of variables visible to the code. A related function is eval, which works very much like exec except that it expects the argument to evaluate to a value, and it returns that value.
1 >>> a = "3*4" 2 >>> b = eval(a) 3 >>> print b 4 12
2.16 import
The real power of Python is in its library modules. They provide a large and consistent set of Application Programming Interfaces (APIs) to many system libraries (often in a way independent of the operating system). For example, if you need to use a random number generator, you can do:
1 >>> import random 2 >>> print random.randint(0, 9) 3 5 This prints a random integer between 0 and 9 (including 9), 5 in the example. The function randint is defined in the module random. It is also possible to import an object from a module into the current namespace:
1 >>> from random import randint 2 >>> print randint(0, 9) or import all objects from a module into the current namespace:
1 >>> from random import * 2 >>> print randint(0, 9) or import everything in a newly defined namespace: 38 THE PYTHON LANGUAGE
1 >>> import random as myrand 2 >>> print myrand.randint(0, 9)
In the rest of this book, we will mainly use objects defined in modules os, sys, datetime, time and cPickle. All of the web2py objects are accessible via a module called gluon, and that is the subject of later chapters. Internally, web2py uses many Python modules (for example thread), but you rarely need to access them directly. In the following subsections we consider those modules that are most useful.
os This module provides an interface to the operating system API. For example:
1 >>> import os 2 >>> os.chdir('..') 3 >>> os.unlink('filename_to_be_deleted')
Some of the os functions, such as chdir, MUST NOT be used in web2py because they are not thread-safe.
os.path.join is very useful; it allows the concatenation of paths in an OS-independent way:
1 >>> import os 2 >>> a = os.path.join('path', 'sub_path') 3 >>> print a 4 path/sub_path System environment variables can be accessed via:
1 >>> print os.environ which is a read-only dictionary.
sys
The sys module contains many variables and functions, but the one we use the most is sys.path. It contains a list of paths where Python searches for modules. When we try to import a module, Python looks for it in all the folders listed in sys.path. If you install additional modules in some location and want Python to find them, you need to append the path to that location to sys.path.
1 >>> import sys 2 >>> sys.path.append('path/to/my/modules') IMPORT 39
When running web2py,Pythonstaysresidentinmemory,andthereisonly one sys.path, while there are many threads servicing the HTTP requests. To avoid a memory leak, it is best to check if a path is already present before appending:
1 >>> path = 'path/to/my/modules' 2 >>> if not path in sys.path: 3 sys.path.append(path)
datetime The use of the datetime module is best illustrated by some examples:
1 >>> import datetime 2 >>> print datetime.datetime.today() 3 2008-07-04 14:03:90 4 >>> print datetime.date.today() 5 2008-07-04 Occasionally you may need to timestamp data based on the UTC time as opposed to local time. In this case you can use the following function:
1 >>> import datetime 2 >>> print datetime.datetime.utcnow() 3 2008-07-04 14:03:90 The datetime modules contains various classes: date, datetime, time and timedelta. The difference between two date or two datetime or two time objects is a timedelta:
1 >>> a = datetime.datetime(2008, 1, 1, 20, 30) 2 >>> b = datetime.datetime(2008, 1, 2, 20, 30) 3 >>> c = b - a 4 >>> print c.days 5 1 In web2py, date and datetime are used to store the corresponding SQL types when passed to or returned from the database.
time The time module differs from date and datetime because it represents time as seconds from the epoch (beginning of 1970).
1 >>> import time 2 >>> t = time.time() 3 1215138737.571 Refer to the Python documentation for conversion functions between time in seconds and time as a datetime. 40 THE PYTHON LANGUAGE
cPickle This is a very powerful module. It provides functions that can serialize almost any Python object, including self-referential objects. For example, let’s build a weird object:
1 >>> class MyClass(object): pass 2 >>> myinstance = MyClass() 3 >>> myinstance.x = 'something' 4 >>> a = [1 ,2, {'hello':'world'}, [3, 4, [myinstance]]] and now:
1 >>> import cPickle 2 >>> b = cPickle.dumps(a) 3 >>> c = cPickle.loads(b)
In this example, b is a string representation of a, and c is a copy of a generated by deserializing b. cPickle can also serialize to and deserialize from a file:
1 >>> cPickle.dumps(a, open('myfile.pickle', 'wb')) 2 >>> c = cPickle.loads(open('myfile.pickle', 'rb')) CHAPTER 3
OVERVIEW
3.1 Startup
web2py comes in binary packages for Windows and Mac OS X. There is also a source code version that runs on Windows, Mac, Linux, and other Unix systems. The Windows and OS X binary versions include the necessary Python interpreter. The source code package assumes that Python is already installed on the computer. web2py requires no installation. To get started, unzip the downloaded zip file for your specific operating system and execute the corresponding web2py file. On Windows, run:
1 web2py.exe On OS X, run:
1 web2py.app On Unix and Linux, run from source by typing:
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 41 Copyright © 2009 42 OVERVIEW
1 python2.5 web2py.py The web2py program accepts various command line options which are discussed later. By default, at startup, web2py displays a startup window:
and then displays a GUI widget that asks you to choose a one-time ad- ministrator password, the IP address of the network interface to be used for the web server, and a port number from which to serve requests. By default, web2py runs its web server on 127.0.0.1:8000 (port 8000 on localhost), but you can run it on any available IP address and port. You can query the IP address of your network interface by opening a command line and typing ipconfig on Windows or ifconfig on OS X and Linux. From now on we assume web2py is running on localhost (127.0.0.1:8000). Use 0.0.0.0:80 to run web2py publicly on any of your network interfaces.
If you do not provide an administrator password, the administration inter- face is disabled. This is a security measure to prevent publicly exposing the admin interface. STARTUP 43
The administration interface is only accessible from localhost unless you run web2py behind Apache with mod proxy. If admin detects a proxy, the session cookie is set to secure and admin login does not work unless the communication between the client and the proxy goes over HTTPS. This is another security measure. All communications between the client and the admin must always be local or encrypted; otherwise an attacker would be able to perform a man-in-the middle attack or a replay attack and execute arbitrary code on the server. After the administration password has been set, web2py starts up the web browser at the page:
1 http://127.0.0.1:8000/
If the computer does not have a default browser, open a web browser and enter the URL.
Clicking on "administrative interface" takes you to the login page for the administration interface. 44 OVERVIEW
The administrator password is the same as the password you chose at startup. Notice that there is only one administrator, and therefore only one administratorpassword. Forsecurity reasons,the developeris asked to choose a new password every time web2py starts unless the
This page lists all installed web2py applications and allows the adminis- trator to manage them. web2py comes with three applications: SAY HELLO 45
• An admin application, the one you are using right now.
• An examples application, with the online interactive documentation and a replica of the web2py official website.
• A welcome application. This is the basic template for any other web2py application. It is referred to as the scaffolding application. This is also the application that welcomes a user at startup. Ready-to-use web2py applications are referred to as web2pyappliances. You can download many freely available appliances from [33]. web2py users are encouraged to submit new appliances, either in open-source or closed-source (compiled and packed) form. From the admin application’s [site] page, you can perform the following operations:
• install an application by completing the form on the bottom right of the page. Give a name to the application, select the file containing a packaged application or the URL where the application is located, and click "submit".
• uninstall an application by clicking the corresponding button. There is a confirmation page.
• create a new application by choosing a name and clicking "submit".
• package an application for distribution by clicking on the correspond- ing button. A downloaded application is a tar file containing everything, including the database. You should never untar this file; it is automati- cally unpackaged by web2py when one installs it using admin.
• clean up an application’s temporary files, such as sessions, errors and cache files.
• EDIT an application.
3.2 Say Hello
Here, as an example, we create a simple web app that displays the message "Hello from MyApp" to the user. We will call this application "myapp". We will also add a counter that counts how many times the same user visits the page. 46 OVERVIEW
You can create a new application simply by typing its name in the form on the top right of the site page in admin.
After you press [submit], the application is created as a copy of the built-in welcome application.
To run the new application, visit:
1 http://127.0.0.1:8000/myapp
Now you have a copy of the welcome application. To edit an application, click on the [EDIT] button for the newly created application. SAY HELLO 47
The EDIT page tells you what is inside the application. Every web2py application consists of certain files, most of which fall into one of five cate- gories: • models: describe the data representation. • controllers: describe the application logic and workflow. • views: describe the data presentation. • languages: describe how to translate the application presentation to other languages. • modules: Python modules that belong to the application. • static files: static images, CSS files [39, 40, 41], JavaScript files [42, 43], etc. Everything is neatly organized following the Model-View-Controller de- sign pattern. Each section in the [EDIT] page corresponds to a subfolder in the application folder. Notice that section headings will toggle their content. Folder names under static files are also collapsible.
Each file listed in the section corresponds to a file physically located in the subfolder. Any operation performed on a file via the admin interface (create, edit, delete) can be performed directly from the shell using your favorite editor. 48 OVERVIEW
The application contains other types of files (database, session files, error files, etc.), but they are not listed on the [EDIT] page because they are not created or modified by the administrator. They are created and modified by the application itself. The controllers contain the logic and workflow of the application. Every URL gets mapped into a call to one of the functions in the controllers (ac- tions). There are two default controllers: "appadmin.py" and "default.py". appadmin provides the database administrative interface; we do not need it now. "default.py" is the controller that you need to edit, the one that is called by default when no controller is specified in the URL. Edit the "index" function as follows:
1 def index(): 2 return "Hello from MyApp" Here is what the online editor looks like:
Save it and go back to the [EDIT] page. Click on the index link to visit the newly created page. When you visit the URL
1 http://127.0.0.1:8000/myapp/default/index the index action in the default controller of the myapp application is called. It returns a string that the browser displays for us. It should look like this: SAY HELLO 49
Now, edit the "index" function as follows:
1 def index(): 2 return dict(message="Hello from MyApp") Also from the [EDIT] page, edit the view default/index (the new file associated with the action) and, in this file, write:
1 2
3 4{{=message}}
5 6Now the action returns a dictionary defining a message. When an ac- tion returns a dictionary, web2py looks for a view with the name "[con- troller]/[function].[extension]" and executes it. Here [extension] is the re- quested extension. If no extension is specified, it defaults to "html", and that is what we will assume here. Under this assumption, the view is an HTML file that embeds Python code using special {{ }} tags. In particular, in the example, the {{=message}} instructs web2py to replace the tagged code with the value of the message returned by the action. Notice that message here is not a web2py keyword but is defined in the action. So far we have not used any web2py keywords. If web2py does not find the requested view, it uses the "generic.html" view that comes with every application.
If an extension other than "html" is specified ("json" for exam- ple), and the view file "[controller]/[function].json"is not found, web2py looks for the view "generic.json". web2py comes with generic.html, generic.json, generic.xml, and generic.rss. These generic views can be modified for each application individually, and additional views can be added easily.
Read more on this topic in Chapter 9. 50 OVERVIEW
If yougobackto[EDIT] andclickonindex,youwill nowseethefollowing HTML page:
3.3 Let’s Count
Let’s now add a counter to this page that will count how many times the same visitor displays the page. web2py automatically and transparently tracks visitors using sessions and cookies. For each new visitor, it creates a session and assigns a unique "session id". The session is a container for variables that are stored server- side. The unique id is sent to the browser via a cookie. When the visitor requestsanotherpagefrom the same application, the browsersendsthe cookie back, it is retrieved by web2py, and the corresponding session is restored. To use the session, modify the default controller:
1 def index(): 2 if not session.counter: 3 session.counter = 1 4 else: 5 session.counter += 1 6 return dict(message="Hello from MyApp", counter=session.counter)
Notice that counter is not a web2py keyword but session is. We are asking web2py to check whether there is a counter variable in the session and,ifnot,tocreateoneandsetitto1. Ifthecounteristhere,weask web2py to increase the counter by 1. Finally we pass the value of the counter to the view. A more compact way to code the same function is this:
1 def index(): 2 session.counter = (session.counter or 0) + 1 3 return dict(message="Hello from MyApp", counter=session.counter) Now modify the view to add a line that displays the value of the counter: SAY MY NAME 51
1 2
3 4{{=message}}
5Number of visits: {{=counter}}
6 7When you visit the index page again (and again) you should get the fol- lowing HTML page:
The counter is associated to each visitor, and is incremented each time the visitor reloads the page. Different visitors see different counters.
3.4 Say My Name
Now create two pages (first and second), where the first page creates a form, asks the visitor’s name, and redirects to the second page, which greets the visitor by name.
form first / second
Write the corresponding actions in the default controller:
1 def first(): 2 return dict() 3 4 def second(): 5 return dict()
Then create a view "default/first.html" for the first action: 52 OVERVIEW
and enter:
1 {{extend 'layout.html'}} 2 What is your name? 3
Finally, create a view "default/second.html" for the second action:
1 {{extend 'layout.html'}} 2
Hello {{=request.vars.visitor_name}}
In both views we have extended the basic "layout.html" view that comes with web2py. The layout view keeps the look and feel of the two pages coherent. The layout file can be edited and replaced easily, since it mainly contains HTML code. If you now visit the first page, type your name: FORM SELF-SUBMISSION 53
and submit the form, you will receive a greeting:
3.5 Form self-submission
The above mechanism for form submission is very common, but it is not good programming practice. All input should be validated and, in the above example, the burden of validation would fall on the second action. Thus the action that performs the validation is different from the action that generated the form. This may cause redundancy in the code. A better pattern for form submission is to submit forms to the same action that generated them, in our example the "first". The "first" action should receive the variables, process them, store them server side, and redirect the visitor to the "second" page, which retrieves the variables. 54 OVERVIEW
redirect first / second You can modify the default controller as follows to implement self-submission:
1 def first(): 2 if request.vars.visitor_name: 3 session.visitor_name = request.vars.visitor_name 4 redirect(URL(r=request, f='second')) 5 return dict() 6 7 def second(): 8 return dict() Accordingly, you need to modify the "default/first.html" view:
1 {{extend 'layout.html'}} 2 What is your name? 3
and the "default/second.html" view needs to retrieve the data from the session instead of from the request.vars:1 {{extend 'layout.html'}} 2
Hello {{=session.visitor_name or "anonymous"}}
From the point of view of the visitor, the self-submission behaves exactly the same as the previous implementation. We have not added validation yet, but it is now clear that validation should be performed by the first action. This approach is better also because the name of the visitor stays in the session, and can be accessed by all actions and views in the applications without having to be passed around explicitly. Note that if the "second"action is ever called before a visitor name is set, it will display "Hello anonymous" because session.visitor name returns None. Alternatively we could have added the following code in the controller (inside or outside the second function:1 if not request.function=='first' and not session.visitor_name: 2 redirect(URL(r=request, f='first')) This is a general mechanism that you can use to enforce authorization on controllers, although see Chapter 8 for a more powerful method. With web2py we can move one step further and ask web2py to generate the form for us, including validation. web2py provides helpers (FORM, INPUT, TEXTAREA, and SELECT/OPTION) with the same names as the equivalentHTMLtags. Theycanbeusedtobuildformseitherinthecontroller or in the view. For example, here is one possible way to rewrite the first action: FORM SELF-SUBMISSION 55
1 def first(): 2 form = FORM(INPUT(_name='visitor_name', requires=IS_NOT_EMPTY()), 3 INPUT(_type='submit')) 4 if form.accepts(request.vars, session): 5 session.visitor_name = form.vars.visitor_name 6 redirect(URL(r=request, f='second')) 7 return dict(form=form)
where we are saying that the FORM tag contains two INPUT tags. The attributes of the input tags are specified by the named arguments starting with underscore. The requires argument is not a tag attribute (because it does not start by underscore) but it sets a validator for the value of visitor name. The form object can be easily serialized in HTML by embedding it in the "default/first.html" view.
1 {{extend 'layout.html'}} 2 What is your name? 3 {{=form}}
The form.accepts method applies the validators. If the self-submitted form passes validation, it stores the variables in the session and redirects as before. If the form does not pass validation, error messages are inserted in the form and shown to the user, shown below:
In the next section we will show how forms canbe generatedautomatically from a model. 56 OVERVIEW
3.6 An Image Blog
Here, as another example, we wish to create a web application that allows the administrator to post images and give them a name, and allows the visitors of the web site to view the images and submit comments. As before, create the new application from the site page in admin and navigate to the [EDIT] page:
We start by creating a model, a representation of the persistent data in the application (the images to upload, their names, and the comments). First, you need to create/edit a model file which, for lack of imagination, we call "db.py". Models and controllers must have a .py extension since they are Python code. If the extension is not provided, it is appended by web2py. Views instead have a .html extension since they mainly contain HTML code.
Edit the "db.py" file by clicking the corresponding "edit" button: AN IMAGE BLOG 57
and enter the following:
1 db = DAL("sqlite://storage.db") 2 3 db.define_table('image', 4 Field('title'), 5 Field('file', 'upload')) 6 7 db.define_table('comment', 8 Field('image_id', db.image), 9 Field('author'), 10 Field('email'), 11 Field('body', 'text')) 12 13 db.image.title.requires = [IS_NOT_EMPTY(), 14 IS_NOT_IN_DB(db, db.image.title)] 15 16 db.comment.image_id.requires = IS_IN_DB(db, db.image.id, '%(title)s') 17 db.comment.author.requires = IS_NOT_EMPTY() 18 db.comment.email.requires = IS_EMAIL() 19 db.comment.body.requires = IS_NOT_EMPTY() 20 21 db.comment.image_id.writable = db.comment.image_id.readable = False Let’s analyze this line by line.
• Line 1 defines a global variable called db that represents the database connection. In this case it is a connection to a SQLite database stored 58 OVERVIEW
in the file "applications/images/databases/storage.db". In the SQLite case, if the database does not exist, it is created. You can change the name of the file, as well as the name of the global variable db, but it is convenient to give them the same name, to make it easy to remember.
• Lines 3-5 define a table "image". define table is a method of the db object. The first argument, "image", is the name of the table we are defining. The other arguments are the fields belonging to that table. This table has a field called "title", a field called "file", and a field called "id" that serves as the table primary key ("id" is not explicitly declared because all tables have an id field by default). The field "title" is a string, and the field "file" is of type "upload". "upload" is a specialtype of field used by the web2py Data Abstraction Layer (DAL) to store the names of uploaded files. web2py knows how to upload files (via streaming if they are large), rename them safely, and store them. When a table is defined, web2py takes one of several possible actions: a)ifthetabledoesnotexist, thetableis created;b)if thetableexistsand does not correspond to the definition, the table is altered accordingly, and if a field has a different type, web2py tries to convert its contents; c) if the table exists and corresponds to the definition, web2py does nothing. This behavior is called "migration". In web2py migrations are auto- matic, but can be disabled for each table by passing migrate=False as the last argument of define table.
• Lines 7-11 define another table called "comment". A comment has an "author", an "email" (we intend to store the email address of the author of the comment), a "body" of type "text" (we intend to use it to store the actual comment posted by the author), and an "image id" field of type reference that points to db.image via the "id" field.
• In lines 13-14 db.image.title represents the field "title" of table "im- age". The attribute requires allows you to set requirements/constraints that will be enforced by web2py forms. Here we require that the "ti- tle" is not empty (IS NOT EMPTY())andthatit is unique(IS NOT IN DB(db, db.image.title)). The objects representing these constraints are called validators. Multiple validators can be grouped in a list. Validators are executed in the order they appear. IS NOT IN DB(a, b) is a special validator that checks that the value of a field b for a new record is not already in a. AN IMAGE BLOG 59
• Line 16 requires that the field "image id" of table "comment" is in db.image.id. As far as the database is concerned, we had already declared this when we defined the table "comment". Now we are explicitly telling the model that this condition should be enforced by web2py, too, at the form processing level when a new comment is posted, so that invalid values do not propagate from input forms to the database. We also require that the "image id" be represented by the "title", ’%(title)s’, of the corresponding record.
• Line 18 indicates that the field "image id" of table "comment" should not be shown in forms, writable=False and not even in readonly forms, readable=False.
The meaning of the validators in lines 17-19 should be obvious. Once a model is defined, if there are no errors, web2py creates an appli- cation administration interface to manage the database. You access it via the "database administration" link in the [EDIT] page or directly:
1 http://127.0.0.1:8000/images/appadmin Here is a screenshot of the appadmin interface:
This interface is coded in the controller called "appadmin.py" and the corresponding view "appadmin.html". From now on, we will refer to this interface simply as appadmin. It allows the administrator to insert new database records, edit and delete existing records, browse tables, and perform database joins. The first time appadmin is accessed, the model is executed and the tables are created. The web2py DAL translates Python code into SQL statements that are specific to the selected database back-end (SQLite in this example). 60 OVERVIEW
You can see the generated SQL from the [EDIT] page by clicking on the "sql.log" link under "models". Notice that the link is not present until the tables have been created.
If youwere to edit the model andaccess appadmin again, web2py would generate SQL to alter the existing tables. The generated SQL is logged into "sql.log". Now go back to appadmin and try to insert a new image record:
web2py has translated the db.image.file "upload" field into an upload form for the file. When the form is submitted and an image file is uploaded, the file is renamed in a secure way that preserves the extension, it is saved with the new name under the application "uploads" folder, and the new name AN IMAGE BLOG 61
is stored in the db.image.file field. This process is designed to prevent directory traversal attacks. When you click on a table name in appadmin, web2py performs a select of all records on the current table, identified by the DAL query
1 db.image.id > 0
and renders the result.
You can select a different set of records by editing the SQL query and pressing "apply". To edit or delete a single record, click on the record id number. 62 OVERVIEW
Becauseof the IS IN DB validator, the reference field "image id" is rendered by a drop-down menu. The items in the drop-down are stored as keys (db.image.id), but are represented by their db.image.title, as specified by the validator. Validators are powerful objects that know how to represent fields, filter field values, generate errors, and format values extracted from the field. The following figure shows what happens when you submit a form that does not pass validation: AN IMAGE BLOG 63
The same forms that are automatically generated by appadmin can also be generated programmatically via the SQLFORM helper and embedded in user applications. These forms are CSS-friendly, and can be customized. Every application has its own appadmin; therefore, appadmin itself can be modified without affecting other applications. So far, the application knows how to store data, and we have seen how to access the database via appadmin. Access to appadmin is restricted to the administrator, and it is not intended as a production web interface for the application; hence the next part of this walk-through. Specifically we want to create:
• An "index" page that lists all available images sorted by title and links to detail pages for the images.
• A "show/[id]" page that shows the visitor the requested image and allows the visitor to view and post comments.
• A "download/[name]" action to download uploaded images. This is represented schematically here:
img index / show/[id] / download/[name] 64 OVERVIEW
Go back to the [EDIT] page and edit the "default.py" controller, replacing its contents with the following:
1 def index(): 2 images = db().select(db.image.ALL, orderby=db.image.title) 3 return dict(images=images) This action returns a dictionary. The keys of the items in the dictionary are interpreted as variables passed to the view associated to the action. If there is no view, the action is rendered by the "generic.html" view that is provided with every web2py application. The index action performs a select of all fields (db.image.ALL) from table image, ordered by db.image.title. The result of the select is a Rows object containing the records. Assign it to a local variable called images returned by the action to the view. images is iterable and its elements are the selected rows. Foreachrowthecolumnscanbeaccessedasdictionaries: images[0][’title’] or equivalently as images[0].title. If you do not write a view, the dictionary is rendered by "views/generic.html" and a call to the index action would look like this:
You have not created a view for this action yet, so web2py renders the set of records in plain tabular form.
Proceed to create a view for the index action. Return to admin, edit "default/index.html" and replace its content with the following:
1 {{extend 'layout.html'}} 2
Current Images
3- 4 {{for image in images:}} 5 {{= LI(A(image.title, _href=URL(r=request, f="show", args=image.id))) }} 6 {{pass}} 7
The first thing to notice is that a view is pure HTML with special {{...}} tags. The code embedded in {{...}} is pure Python code with one caveat: indentation is irrelevant. Blocks of code start with lines ending in colon (:) and end in lines beginning with the keyword pass. In some cases the end of a block is obvious from context and the use of pass is not required. Lines 5-7 loop over the image rows and for each row image display:
1 LI(A(image.title, _href=URL(r=request, f='show', args=image.id))
This is a
1 URL (r=request, f='show', args=image.id)
i.e., the URL within the same application and controller as the current request r=request, calling the function called "show", f="show", and passing a single argument to the function, args=image.id. LI, A, etc. are web2py helpers that map to the corresponding HTML tags. Their unnamed arguments are interpreted as objects to be serialized and inserted in the tag’s innerHTML. Named arguments starting with an underscore (for example href) are interpreted as tag attributes but without the underscore. For example href is the href attribute, class is the class attribute, etc. As an example, the following statement:
1 {{=LI(A('something', _href=URL(r=request, f='show', args=123))}}
is rendered as:
1
Ahandfulofhelpers(INPUT, TEXTAREA, OPTION and SELECT)alsosupportsome special named attributes not starting with underscore (value, and requires). They are important for building custom forms and will be discussed later. Go back to the [EDIT] page. It now indicates that "default.py exposes index". By clicking on "index", you can visit the newly created page:
1 http://127.0.0.1:8000/images/default/index
which looks like: 66 OVERVIEW
If you click on the image name link, you are directed to:
1 http://127.0.0.1:8000/images/default/show/1 and this results in an error, since you have not yet created an action called "show" in controller "default.py". Let’s edit the "default.py" controller and replace its content with:
1 def index(): 2 images = db().select(db.image.ALL, orderby=db.image.title) 3 return dict(images=images) 4 5 def show(): 6 image = db(db.image.id==request.args(0)).select()[0] 7 form = SQLFORM(db.comment) 8 form.vars.image_id = image.id 9 if form.accepts(request.vars, session): 10 response.flash = 'your comment is posted' 11 comments = db(db.comment.image_id==image.id).select() 12 return dict(image=image, comments=comments, form=form) 13 14 def download(): 15 return response.download(request, db) The controller contains two actions: "show" and "download". The "show" action selects the image with the id parsed from the request args and all comments related to the image. "show" then passes everything to the view "default/show.html". The image id referenced by:
1 URL (r=request, f='show', args=image.id)}
in "default/index.html", can be accessed as: request.args(0) from the "show" action. The "download" action expects a filename in request.args(0), builds a path to the location where that file is supposed to be, and sends it back to the client. If the file is too large, it streams the file without incurring any memory overhead. AN IMAGE BLOG 67
Notice the following statements:
• Line 7 creates an insert form SQLFORM for the db.comment table using only the specified fields.
• Line 8 sets the value for the reference field, which is not part of the input form because it is not in the list of fields specified above.
• Line 9 processes the submitted form (the submitted form variables are in request.vars) within the current session (the session is used to prevent double submissions, and to enforce navigation). If the submitted form variables are validated, the new comment is inserted in the db.comment table; otherwise the form is modified to include error messages (for example, if the author’s email address is invalid). This is all done in line 9!.
• Line 10 is only executed if the form is accepted, after the record is inserted into the database table. response.flash is a web2py vari- able that is displayed in the views and used to notify the visitor that something happened.
• Line 11 selects all comments that reference the current image.
The "download" action is already defined in the "default.py" controller of the scaffolding application.
The "download" action does not return a dictionary, so it does not need a view. The "show" action, though, should have a view, so return to admin and create a new view called "default/show.html" by typing "default/show" in the create view form: 68 OVERVIEW
Edit this new file and replace its content with the following:
1 {{extend 'layout.html'}} 2
Image: {{=image.title}}
3Comments
9 {{for comment in comments:}} 10
{{=comment.author}} says {{=comment.body}}
11 {{pass}} 12 {{else:}} 13No comments posted yet
14 {{pass}} 15Post a comment
16 {{=form}}This view displays the image.file by calling the "download" action inside an tag. If there are comments, it loops over them and displays each one.
Here is how everything will appear to a visitor. ADDING CRUD 69
When a visitor submits a comment via this page, the comment is stored in the database and appended at the bottom of the page.
3.7 Adding CRUD
web2py also provides a CRUD (Create/Read/Update/Delete) API that sim- plifies forms even more. To use CRUD it is necessaryto define it somewhere, such as in module "db.py":
1 from gluon.tools import Crud 2 crud = Crud(globals(), db)
These two lines are already in the scaffolding application.
The crud object provides high-level methods, for example:
1 form = crud.create(...) that can be used to replace the programming pattern:
1 form = SQLFORM(...) 2 if form.accepts(...): 3 session.flash = ... 4 redirect(...) 70 OVERVIEW
Here, we rewrite the previous "show" action using crud:
1 def show(): 2 image = db(db.image.id==request.args(0)).select()[0] 3 db.comment.image_id.default = image.id 4 form = crud.create(db.image, next=URL(r=request, args=image.id), 5 message='your comment is posted') 6 comments = db(db.comment.image_id==image.id).select() 7 return dict(image=image, comments=comments, form=form)
The next argumentof crud.create is the URL to redirect to after the form is accepted. The message argument is the one to be displayed upon acceptance. You can read more about CRUD in Chapter 7.
3.8 Adding Authentication
The web2py API for Role-Based Access Control is quite sophisticated, but for now we will limit ourselves to restricting access to the show action to authenticated users, deferring a more detailed discussion to Chapter 8. To limit access to authenticated users, we need to complete three steps. In a model, for example "db.py", we need to add:
1 from gluon.tools import Auth 2 auth = Auth(globals(), db) 3 auth.define_tables()
In our controller, we need to add one action:
1 def user(): 2 return dict(form=auth())
Finally, we decorate the functions that we want to restrict, for example:
1 @auth.requires_login() 2 def show(): 3 image = db(db.image.id==request.args(0)).select()[0] 4 db.comment.image_id.default = image.id 5 form = crud.create(db.image, next=URL(r=request, args=image.id), 6 message='your comment is posted') 7 comments = db(db.comment.image_id==image.id).select() 8 return dict(image=image, comments=comments, form=form)
Any attempt to access
1 http://127.0.0.1:8000/images/default/show/[image_id]
will require login. If the user is not logged it, the user will be redirected to
1 http://127.0.0.1:8000/images/default/user/login A WIKI 71
The user functionalsoexposes,amongothers,thefollowingactions:
1 http://127.0.0.1:8000/images/default/user/logout 2 http://127.0.0.1:8000/images/default/user/register 3 http://127.0.0.1:8000/images/default/user/profile 4 http://127.0.0.1:8000/images/default/user/change_password
Now, a first time user needs to register in order to be able to login and read/post comments.
Both the auth object and the user function are already defined in the scaffolding application. The auth object is highly customiz- able and can deal with email verification, registration approvals, CAPTCHA, and alternate login methods via plugins.
3.9 A Wiki
In this section, we build a wiki. The visitor will be able to create pages, search them (by title), and edit them. The visitor will also be able to post comments (exactly as in the previous applications), and also post documents (as attachments to the pages) and link them from the pages. As a convention, we adopt the Markdown syntax for our wiki syntax. We will also implement a search page with Ajax, an RSS feed for the pages, and a handler to search the pages via XML-RPC [44]. The following diagram lists the actions that we need to implement and the links we intend to build among them. 72 OVERVIEW
index / create OO OOO OOO OOO ' img search show/[id] / download/[name] R pp8 RRR ppp RRR ajax pp RRR pp RRR pp ( bg find edit/[id] documents/[id]
Start by creating a new scaffolding app, naming it "mywiki". The model must contain three tables: page, comment, and document. Both comment and document reference page because they belong to page. A document contains a file field of type upload as in the previous images application. Here is the complete model:
1 db = DAL('sqlite://storage.db') 2 3 from gluon.tools import * 4 auth = Auth(globals(),db) 5 auth.define_tables() 6 crud = Crud(globals(),db) 7 8 if auth.is_logged_in(): 9 user_id = auth.user .id 10 else: 11 user_id = None 12 13 db.define_table('page', 14 Field('title'), 15 Field('body', 'text'), 16 Field('created_on', 'datetime', default=request.now), 17 Field('created_by', db.auth_user, default=user_id)) 18 19 db.define_table('comment', 20 Field('page_id', db.page), 21 Field('body', 'text'), 22 Field('created_on', 'datetime', default=request.now), 23 Field('created_by', db.auth_user, default=user_id)) 24 25 db.define_table('document', 26 Field('page_id', db.page), 27 Field('name'), 28 Field('file', 'upload'), 29 Field('created_on', 'datetime', default=request.now), 30 Field('created_by', db.auth_user, default=user_id)) 31 32 db.page.title.requires = [IS_NOT_EMPTY(), IS_NOT_IN_DB(db, 'page. title')] 33 db.page.body.requires = IS_NOT_EMPTY() 34 db.page.created_by.readable = False A WIKI 73
35 db.page.created_by.writable = False 36 db.page.created_on.readable = False 37 db.page.created_on.writable = False 38 39 db.comment.page_id.requires = IS_IN_DB(db, 'page.id', '%(title)s') 40 db.comment.body.requires = IS_NOT_EMPTY() 41 db.comment.page_id.readable = False 42 db.comment.page_id.writable = False 43 db.comment.created_by.readable = False 44 db.comment.created_by.writable = False 45 db.comment.created_on.readable = False 46 db.comment.created_on.writable = False 47 48 db.document.page_id.requires = IS_IN_DB(db, 'page.id', '%(title)s') 49 db.document.name.requires = [IS_NOT_EMPTY(), IS_NOT_IN_DB(db, ' document.name')] 50 db.document.page_id.readable = False 51 db.document.page_id.writable = False 52 db.document.created_by.readable = False 53 db.document.created_by.writable = False 54 db.document.created_on.readable = False 55 db.document.created_on.writable = False Edit the controller "default.py" and create the following actions: • index: list all wiki pages • create: post another wiki page • show: show a wiki page and its comments, and append comments • edit: edit an existing page • documents: manage the documents attached to a page • download: download a document (as in the images example) • search: display a search box and, via an Ajax callback, return all matching titles as the visitor types
• bg find: the Ajax callback function. It returns the HTML that gets embedded in the search page while the visitor types. Here is the "default.py" controller:
1 def index(): 2 """ this controller returns a dictionary rendered by the view 3 it lists all wiki pages 4 >>> index().has_key('pages') 5 True 6 """ 7 pages = db().select(db.page.id, db.page.title, 8 orderby=db.page.title) 9 return dict(pages=pages) 74 OVERVIEW
10 11 @auth.requires_login() 12 def create(): 13 "creates a new empty wiki page" 14 form = crud.create(db.page, next = URL(r=request, f='index')) 15 return dict(form=form) 16 17 def show(): 18 "shows a wiki page" 19 thispage = db.page[request.args(0)] 20 if not thispage: 21 redirect(URL(r=request, f='index')) 22 db.comment.page_id.default = thispage.id 23 if user_id: 24 form = crud.create(db.comment) 25 else: 26 form = None 27 pagecomments = db(db.comment.page_id==thispage.id).select() 28 return dict(page=thispage, comments=pagecomments, form=form) 29 30 @auth.requires_login() 31 def edit(): 32 "edit an existing wiki page" 33 thispage = db.page[request.args(0)] 34 if not thispage: 35 redirect(URL(r=request, f='index')) 36 form = crud.update(db.page, thispage, 37 next = URL(r=request, f='show', args=request.args)) 38 return dict(form=form) 39 40 @auth.requires_login() 41 def documents(): 42 "lists all documents attached to a certain page" 43 thispage = db.page[request.args(0)] 44 if not thispage: 45 redirect(URL(r=request, f='index')) 46 db.document.page_id.default = thispage.id 47 form = crud.create(db.document) 48 pagedocuments = db(db.document.page_id==thispage.id).select() 49 return dict(page=thispage, documents=pagedocuments, form=form) 50 51 def user(): 52 return dict(form=auth()) 53 54 def download(): 55 "allows downloading of documents" 56 return response.download(request, db) 57 58 def search(): 59 "an ajax wiki search page" 60 return dict(form=FORM(INPUT(_id='keyword', 61 _onkeyup="ajax('bg_find', ['keyword'], 'target');")), 62 target_div=DIV(_id='target')) 63 64 def bg_find(): 65 "an ajax callback that returns a
- of links to wiki pages" A WIKI 75
- <hello>world
- ... tags.
- <hello>
- world
- <hello>
- world ul> 142 THE VIEWS
- {{=fieldname}} error: {{=form.errors[fieldname]}} 6 {{pass}} 7
66 pattern = '%' + request.vars.keyword.lower() + '%' 67 pages = db(db.page.title.lower().like(pattern))\ 68 .select(orderby=db.page.title) 69 items = [A(row.title, _href=URL(r=request, f=show, args=row.id)) \ 70 for row in pages] 71 return UL(*items).xml()
Lines 2-6 provide a comment for the index action. Lines 4-5 inside the comment are interpreted by python as test code (doctest). Tests can be run via the admin interface. In this case the tests verify that the index action runs without errors. Lines 19, 33, and 43 try fetch a page record with the id in request.args(0). Line 14, 24 and 47 define and process create forms, for a new page and a new comment and a new document respectively. Line 36 defines and process an update form for a wiki page. Some magic happens in line 59. The onkeyup attribute of the INPUT tag "keyword" is set. Every time the visitor presses a key or releases a key, the JavaScript code inside the onkeyup attribute is executed, client-side. Here is the JavaScript code:
1 ajax('bg_find',['keyword'], 'target');
ajax is a JavaScript function defined in the file "web2py ajax.html" which is included by the default "layout.html". It takes three parameters: the URL of the action that performs the synchronous callback ("bg find"), a list of the IDs of variables to be sent to the callback (["keyword"]), and the ID where the response has to be inserted ("target"). As soon as you type something in the search box and release a key, the client calls the server and sends the content of the ’keyword’ field, and, when the sever responds, the response is embedded in the page itself as the innerHTML of the ’target’ tag. The ’target’ tag is a DIV defined in line 75. It could have been defined in the view as well. Here is the code for the view "default/create.html":
1 {{extend 'layout.html'}} 2
Create new wiki page
3 {{=form}}If you visit the create page, you see the following: 76 OVERVIEW
Here is the code for the view "default/index.html":
1 {{extend 'layout.html'}} 2
Available wiki pages
3 [ {{=A('search', _href=URL(r=request, f='search'))}} ]4
- {{for page in pages:}} 5 {{= LI(A(page.title, _href=URL(r=request, f='show', args=page.id) ))}} 6 {{pass}}
Here is the code for the view "default/show.html":
1 {{extend 'layout.html'}} 2
{{=page.title}}
A WIKI 773 [ {{=A('edit', _href=URL(r=request, f='edit', args=request.args))}} 4 | {{=A('documents', _href=URL(r=request, f='documents', args=request. args))}} ]
5 {{import gluon.contrib.markdown}} 6 {{=gluon.contrib.markdown.WIKI(page.body)}} 7
Comments
8 {{for comment in comments:}} 9{{=db.auth_user[comment.created_by].first_name}} on {{=comment. created_on}} 10 says {{=comment.body}}
11 {{pass}} 12Post a comment
13 {{=form}}web2py includes gluon.contrib.markdown.WIKI,whichknowshowtocon- vert Markdown syntax to HTML. Alternatively, you could have chosen to accept raw HTML instead of Markdown syntax. In this case you would have to replace:
1 {{=gluon.contrib.markdown.WIKI(page.body)}}
with:
1 {{= XML(page.body)}}
(so that the XML does not get escaped, as by default web2py behavior). This can be done better with:
1 {{= XML(page.body, sanitize=True)}}
By setting sanitize=True, you tell web2py to escape unsafe XML tags such as "') 2 Unescaped executable input such as this (for example, entered in the body of a comment in a blog) is unsafe, because it can be used to generate Cross Site Scripting (XSS) attacks against other visitors to the page. The web2py XML helper can sanitize our text to prevent injections and escape all tags except those that you explicitly allow. Here is an example:
1 >>> print XML('', sanitize=True) 2 <script>alert("unsafe!")</script>
The XML constructors, by default, consider the content of some tags and some of their attributes safe. You can override the defaults using the optional permitted tags and allowed attributes arguments. Herearethedefaultvalues of the optional arguments of the XML helper.
1 XML(text, sanitize=False, 2 permitted_tags=['a', 'b', 'blockquote', 'br/', 'i', 'li', 3 'ol', 'ul', 'p', 'cite', 'code', 'pre', 'img/'], 4 allowed_attributes={'a':['href', 'title'], 5 'img':['src', 'alt'], 'blockquote':['type']})
Built-in Helpers
A This helper is used to build links.
1 >>> print A('
B This helper makes its contents bold.
1 >>> print B('
BODY This helper makes the body of a page.
1 >>> print BODY('
CENTER This helper centers its content.
1 >>> print CENTER('
CODE This helper performs syntax highlighting for Python, C, C++, HTML and web2py code, and is preferable to PRE for code listings. CODE also has the ability to create links to the web2py API documentation. Here is an example of highlighting sections of Python code.
1 >>> print CODE('print "hello"', language='python').xml() 2
< pre style=" 3 font-size: 11px; 4 font-family: Bitstream Vera Sans Mono,monospace; 5 background-color: transparent; 6 margin: 0; 7 padding: 5px; 8 border: none; 9 background-color: #E0E0E0; 10 color: #A0A0A0; 11 ">1. | print < span style="color: #FF9966">"hello" |
< pre style=" 5 .... 6 "><html><body>{{=request.env. remote_add}}</body< span style="font-weight: bold">></html> |
These are the default arguments for the CODE helper:
1 CODE("print 'hello world'", language='python', link=None, counter=1, styles={})
Supported values for the language argument are "python", "html plain", "c", "cpp", "web2py", and "html". The "html" language interprets {{ and }} tags as "web2py" code, while "html plain" doesn’t. If a link value is specified, for example "/examples/global/vars/", web2py API references in the code are linked to documentation at the link URL. For 136 THE VIEWS
example "request" would be linked to "/examples/global/vars/request". In the above example, the link URL is handled by the "var" action in the "global.py" controller that is distributed as part of the web2py "examples" application. The counter argument is used for line numbering. It can be set to any of three different values. It can be None for no line numbers, a numerical value specifying the start number, or a string. If the counter is set to a string, it is interpreted as a prompt, and there are no line numbers.
DIV All helpers apart from XML are derived from DIV and inherit its basic methods.
1 >>> print DIV('
EM Emphasizes its content.
1 >>> print EM('
FIELDSET This is used to create an input field together with its label.
1 >>> print FIELDSET('Height:', INPUT(_name='height'), _class='test') 2
FORM This is one of the most important helpers. In its simple form, it just makes a
tag, but because helpers are objects and have knowledge of what they contain, they can process submitted forms (for example, perform validation of the fields). This will be discussed in detail in Chapter 7.1 >>> print FORM(INPUT(_type='submit'), _action='', _method='post') 2
The "enctype" is "multipart/form-data" by default. The constructor of a FORM, and of SQLFORM, can also take a special argument called hidden. When a dictionary is passed as hidden, its items are translated into "hidden" INPUT fields. For example:1 >>> print FORM(hidden=dict(a='b')) 2
H1, H2, H3, H4, H5, H6 These helpers are for paragraph headings and subheadings:
1 >>> print H1('<hello>world
HTML HELPERS 137
HEAD For tagging the HEAD of an HTML page.
1 >>> print HEAD(TITLE('
HTML This helper is a little different. In addition to making the tags, it prepends the tag with a doctype string [49, 50, 51].
1 >>> print HTML(BODY('
1 HTML(..., lang='en', doctype='transitional') where doctype can be ’strict’, ’transitional’, ’frameset’, ’html5’, or a full doctype string.
XHTML XHTML is similar to HTML but it creates an XHTML doctype instead.
1 XHTML(..., lang='en', doctype='transitional', xmlns='http://www.w3. org/1999/xhtml') where doctype can be ’strict’, ’transitional’, ’frameset’, or a full doctype string.
INPUT Creates an
1 >>> print INPUT(_name='test', _value='a') 2 It also takes an optional special argument called "value", distinct from " value". The latter sets the default value for the input field; the former sets its current value. For an input of type "text", the former overrides the latter:
1 >>> print INPUT(_name='test', _value='a', value='b') 2
For radio buttons INPUT selectively sets the "checked" attribute:
1 >>> for v in ['a', 'b', 'c']: 2 >>> print INPUT(_type='radio', _name='test', _value=v, value='b') , v 3 a 4 b 5 c and similarly for checkboxes: 138 THE VIEWS
1 >>> print INPUT(_type='checkbox', _name='test', _value='a', value= True) 2 3 >>> print INPUT(_type='checkbox', _name='test', _value='a', value= False) 4
IFRAME This helper includes another web page in the current page. The url of the other page is specified via the " src" attribute.
1 >>> print IFRAME(_src='http://www.web2py.com') 2
LABEL It is used to create a LABEL tag for an INPUT field.
1 >>> print LABEL('
LI It makes a list item and should be contained in a UL or OL tag.
1 >>> print LI('
LEGEND It is used to create a legend tag for a field in a form.
1 >>> print LEGEND('Name', _for='somefield') 2
META Tobeusedforbuilding METAtagsin theHTML head. Forexample:
1 >>> print META(_name='security', _content='high') 2
OBJECT Usedto embedobjects(for example, a flashplayer)in the HTML.
1 >>> print OBJECT('
OL It standsfor Ordered List. The list shouldcontain LI tags. OL arguments that are not LI objects are automatically enclosed in
1 >>> print OL('
ON This is here for backward compatibility and it is simply an alias for True. It is used exclusively for checkboxes and deprecated since True is more Pythonic.
1 >>> print INPUT(_type='checkbox', _name='test', _checked=ON) 2 HTML HELPERS 139
OPTION This should only be used as part of a SELECT/OPTION combi- nation.
1 >>> print OPTION('
Asin thecaseof INPUT, web2py make a distinction between " value" (the value of the OPTION), and "value" (the current value of the enclosing select). If they are equal, the option is "selected".
1 >>> print SELECT('a', 'b', value='b'): 2
P This is for tagging a paragraph.
1 >>> print P(' <hello>world
PRE Generates a
...tag for displaying preformatted text. The CODE helper is generally preferable for code listings.
1 >>> print PRE('<hello>world
SCRIPT This is include or link a script, such as JavaScript. The content between the tags is rendered as an HTML comment, for the benefit of really old browsers.
1 >>> print SCRIPT('alert("hello world");', _language='javascript') 2
SELECT Makes a tag. This is used with the OPTION helper. Those SELECT arguments that are not OPTION objects are automatically converted to options.
1 >>> print SELECT('
SPAN Similar to DIV but used to tag inline (rather than block) content.
1 >>> print SPAN('
STYLE Similar to script, but used to either include or link CSS code. Here the CSS is included:
1 >>> print STYLE(XML('body {color: white}')) 2 and here it is linked:
1 >>> print STYLE(_src='style.css') 2
TABLE, TR, TD These tags (along with the optional THEAD, TBODY and TFOOTER helpers) are used to build HTML tables.
1 >>> print TABLE(TR(TD('a' ), TD('b')),D TR(T ('c'), TD('d'))) 2
a | b | ||||||
c | d |
a | b | ||||||
c | d |
a | b | ||||||
c | d |
a | b | ||||||
c | d | ||||||
<hello> | |||||||
<hello> | <hello>world | ||||||
---|---|---|---|---|---|---|---|
<hello> | ... | tags. TR arguments that are not TD objects will be automatically converted.||||||
<hello> | world | tr>
a | : | hello world |
b | : | 1 2 |
5.4 Page Layout
Views can extend and include other views in a tree-like structure, as in the following example (an upward arrow means extend, while a downward arrow means include):
layout.html Q XXX mmm O QQQ XXXXX mmm QQQ XXXXX mmm QQQ XXXXX mm QQ XXXXX vmm Q( X+ header.html index.html sidebar.html footer.html
body.html In this example, the view "index.html" extends "layout.html" and includes "body.html". "layout.html" includes "header.html", "sidebar.html" and "footer.html". Therootofthe tree iswhatwecallalayoutview. Justlike anyotherHTML template file, you can edit it using the web2py administrative interface. The file name "layout.html" is just a convention. Here is a minimalist page that extends the "layout.html" view and includes the "page.html" view:
1 {{extend 'layout.html'}} 2
Hello World
3 {{include 'page.html'}} 144 THE VIEWSThe extended layout file must contain an {{include}} directive, something like:
1
extend and include are special template directives, not Python commands.
Layouts are used to encapsulate page commonality (headers, footers, menus), and though they are not mandatory, they will make your applica- tion easier to write and maintain. In particular, we suggest writing layouts that take advantage of the following variables that can be set in the controller. Using these well known variables will help make your layouts interchange- able:
1 response.title 2 response.subtitle 3 response.author 4 response.keywords 5 response.description 6 response.flash 7 response.menu These are all strings and their meaning should be obvious, except for response.menu. The response.menu menu is a list of three-element tuples. The three elements are: the link name, a boolean representing whether the link is active (is the current link), and the URL of the linked page. For example:
1 response.menu = [['Google', False', 'http://www.google.com'], 2 ['Index', True, URL(r=request, f='index')]] We also recommend that you use:
1 {{include 'web2py_ajax.html'}} in the HTML head, since this will include the jQuery libraries and define some backward-compatible JavaScript functions for special effects and Ajax. Here is a minimal "layout.html" page based on the preceding recommen- dations:
1 2 PAGE LAYOUT 145
3
4 5 6 7 8 9 10 11Storing the original filename web2py automatically stores the original filename inside the new UUID filename and retrieves it when the file is downloaded. Upon download, the original filename is stored in the content-disposition header of the HTTP response. This is all done transparently without the need for programming. Occasionally you may want to store the original filename in a database field. In this case, you need to modify the model and add a field to store it in:
1 db.define_table('person', 2 Field('name', requires=IS_NOT_EMPTY()), 3 Field('image_filename'), 4 Field('image', 'upload')) 198 FORMS AND VALIDATORS
then you need to modify the controller to handle it:
1 def display_form(): 2 if len(request.args): 3 records = db(db.person.id==request.args[0]).select() 4 if len(request.args) and len(records): 5 url = URL(r=request, f='download') 6 form = SQLFORM(db.person, records[0], deletable=True, 7 upload=url, fields=['name', 'image']) 8 else: 9 form = SQLFORM(db.person, fields=['name', 'image']) 10 if request.vars.image: 11 form.vars.image_filename = request.vars.image.filename 12 if form.accepts(request.vars, session): 13 response.flash = 'form accepted' 14 elif form.errors: 15 response.flash = 'form has errors' 16 return dict(form=form)
Notice that the SQLFORM does not display the "image filename" field. The "display form" action moves the filename of the request.vars.image into the form.vars.image filename, so that it gets processed by accepts and stored in the database. The download function, before serving the file, checks in the database for the original filename and uses it in the content-disposition header.
Removing the action file
The SQLFORM, upon deleting a record, does not delete the physical uploaded file(s) referenced by the record. The reason is that web2py does not know whether the same file is used/linked by other tables or used for other purpose. If you know it is safe to delete the actual file when the corresponding record is deleted, you can do the following:
1 db.define_table('image', 2 Field('name'), 3 Field('file','upload',autodelete=True))
The autodelete attribute is False by default. When set to True is makes sure the file is deleted when the record is deleted.
Links to referencing records Now consider the case of two tables linked by a reference field. For example:
1 db.define_table('person', 2 Field('name', requires=IS_NOT_EMPTY())) 3 db.define_table('dog', 4 Field('owner', db.person), SQLFORM 199
5 Field('name', requires=IS_NOT_EMPTY())) 6 db.dog.owner.requires = IS_IN_DB(db,db.person.id,'%(name)s') A person has dogs, and each dog belongs to an owner, which is a person. The dog owner is required to reference a valid db.person.id by ’%(name)s’. Let’s use the appadmin interface for this application to add a few persons and their dogs. When editing an existing person, the appadmin UPDATE form shows a link to a page that lists the dogs that belong to the person. This behavior can be replicated using the linkto argument of the SQLFORM. linkto has to point to the URL of a new action that receives a query string from the SQLFORM and lists the corresponding records. Here is an example:
1 def display_form(): 2 if len(request.args): 3 records = db(db.person.id==request.args[0]).select() 4 if len(request.args) and len(records): 5 url = URL(r=request, f='download') 6 link = URL(r=request, f='list_records') 7 form = SQLFORM(db.person, records[0], deletable=True, 8 upload=url, linkto=link) 9 else: 10 form = SQLFORM(db.person) 11 if form.accepts(request.vars, session): 12 response.flash = 'form accepted' 13 elif form.errors: 14 response.flash = 'form has errors' 15 return dict(form=form)
Here is the page:
There is a link called "dog.owner". The name of this link can be changed via the labels argument of the SQLFORM, for example:
1 labels = {'dog.owner':"This person's dogs"} 200 FORMS AND VALIDATORS
If you click on the link you get directed to:
1 /test/default/list_records/dog?query=dog.owner%3D5
"list records" is the specified action, with request.args[0] set to the name of the referencing table and request.vars.query set to the SQL query string. The query string in the URL contains the value "dog.owner=5" appropriately url-encoded (web2py decodes this automatically when the URL is parsed). You can easily implement a very general "list records" action as follows:
1 def list_records(): 2 table = request.args[0] 3 query = request.vars.query 4 records = db(query).select(db[table].ALL) 5 return dict(records=records) with the associated "default/list records.html" view:
1 {{extend 'layout.html'}} 2 {{=records}} When a set of records is returned by a select and serialized in a view, it is first converted into a SQLTABLE object (not the same as a Table) and then serialized into an HTML table, where each field corresponds to a table column.
Prepopulating the form It is always possible to prepopulate a form using the syntax:
1 form.vars.name = 'fieldvalue' Statements like the one above must be inserted after the form declaration and before the form is accepted, whether or not the field ("name" in the example) is explicitly visualized in the form.
SQLFORM without database IO Thereare times whenyou want to generatea form from a databasetable using SQLFORM and you want to validate a submitted form accordingly, but you do not want any automatic INSERT/UPDATE/DELETE in the database. This is the case, for example, when one of the fields needs to be computed from the value of other input fields. This is also the case when you need to perform additional validation on the inserted data that cannot be achieved via standard validators. This can be done easily by breaking: SQLFORM.FACTORY 201
1 form = SQLFORM(db.person) 2 if form.accepts(request.vars, session): 3 response.flash = 'record inserted' into:
1 form = SQLFORM(db.person) 2 if form.accepts(request.vars, session, dbio=False): 3 ### deal with uploads explicitly 4 form.vars.id = db.person.insert(**dict(form.vars)) 5 response.flash = 'record inserted' The same can be done for UPDATE/DELETE forms by breaking:
1 form = SQLFORM(db.person,record) 2 if form.accepts(request.vars, session): 3 response.flash = 'record updated' into:
1 form = SQLFORM(db.person,record) 2 if form.accepts(request.vars, session, dbio=False): 3 if form.vars.get('delete_this_record', False): 4 db(db.person.id==record.id).delete() 5 else: 6 record.update_record(**dict(form.vars)) 7 response.flash = 'record updated' In both cases web2py deals with the storage and renaming of the uploaded file as if dbio=True, the defaul scenario. The uploaded filename is in:
1 form.vars['%s_newfilename' % fieldname] For more details, refer to the source code in "gluon/sqlhtml.py".
7.3 SQLFORM.factory
Therearecaseswhenyouwantto generateforms as if youhadadatabasetable but you do not want the database table. You simply want to take advantage of the SQLFORM capability to generate a nice looking CSS-friendly form and perhaps perform file upload and renaming. This can be done via a form factory. Here is an example where you generate the form, perform validation, upload a file and store everything in the session :
1 def form_from_factory() 2 form = SQLFORM.factory( 3 Field('your_name', requires=IS_NOT_EMPTY()), 4 Field('your_image')) 5 if form.accepts(request.vars, session): 6 response.flash = 'form accepted' 7 session.your_name = form.vars.your_name 202 FORMS AND VALIDATORS
8 session.filename = form.vars.your_image 9 elif form.errors: 10 response.flash = 'form has errors' 11 return dict(form=form)
Here is the "default/form from factory.html" view:
1 {{extend 'layout.html'}} 2 {{=form}}
You need to use an underscore instead of a space for field labels, or explicitly pass a dictionary of labels to form factory, as you would for a SQLFORM.
7.4 Validators
Validators are classes used to validate input fields (including forms generated from database tables). Here is an example of using a validator with a FORM:
1 INPUT(_name='a', requires=IS_INT_IN_RANGE(0, 10))
Here is an example of how to require a validator for a table field:
1 db.define_table('person', Field('name')) 2 db.person.name.requires = IS_NOT_EMPTY()
Validators are always assigned using the requires attribute of a field. A field can have a single validator or multiple validators. Multiple validators are made part of a list:
1 db.person.name.requires = [IS_NOT_EMPTY(), 2 IS_NOT_IN_DB(db, 'person.name')]
Validators are called by the function accepts on a FORM or other HTML helper object that contains a form. They are called in the order in which they are listed. Built-in validators have constructors that take the optional argument error message, which allows you to override the default error message. Here is an example of a validator on a database table:
1 db.person.name.requires = IS_NOT_EMPTY(error_message=T('fill this!'))
wherewe haveusedthe translation operator T to allow for internationalization. Notice that default error messages are not translated. VALIDATORS 203
Basic Validators
IS ALPHANUMERIC This validator checks that a field value contains only characters in the ranges a-z, A-Z, or 0-9.
1 requires = IS_ALPHANUMERIC(error_message=T('must be alphanumeric!'))
IS DATE This validator checks that a field value contains a valid date in the specified format. It is good practice to specify the format using the translation operator, in order to support different formats in different locales.
1 requires = IS_DATE(format=T('%Y-%m-%d'), 2 error_message=T('must be YYYY-MM-DD!')) For the full description on % directives look under the IS DATETIME val- idator.
IS DATETIME This validator checks that a field value contains a valid datetime in the specified format. It is good practice to specify the format using the translation operator, in order to support different formats in different locales.
1 requires = IS_DATETIME(format=T('%Y-%m-%d %H:%M:%S'), 2 error_message=T('must be YYYY-MM-DD HH:MM:SS!' )) The following symbols can be used for the format string:
1 %a Locale's abbreviated weekday name. 2 %A Locale's full weekday name. 3 %b Locale's abbreviated month name. 4 %B Locale's full month name. 5 %c Locale's appropriate date and time representation. 6 %d Day of the month as a decimal number [01,31]. 7 %H Hour (24-hour clock) as a decimal number [00,23]. 8 %I Hour (12-hour clock) as a decimal number [01,12]. 9 %j Day of the year as a decimal number [001,366]. 10 %m Month as a decimal number [01,12]. 11 %M Minute as a decimal number [00,59]. 12 %p Locale's equivalent of either AM or PM. 13 %S Second as a decimal number [00,61]. 14 %U Week number of the year (Sunday as the first day of the week) 15 as a decimal number [00,53]. All days in a new year preceding 16 the first Sunday are considered to be in week 0. 17 %w Weekday as a decimal number [0(Sunday),6]. 18 %W Week number of the year (Monday as the first day of the week) 19 as a decimal number [00,53]. All days in a new year preceding 20 the first Monday are considered to be in week 0. 21 %x Locale's appropriate date representation. 22 %X Locale's appropriate time representation. 23 %y Year without century as a decimal number [00,99]. 24 %Y Year with century as a decimal number. 25 %Z Time zone name (no characters if no time zone exists). 26 %% A literal "%" character. 204 FORMS AND VALIDATORS
IS EMAIL It checks that the field value looks like an email address. It does not try to send email to confirm.
1 requires = IS_EMAIL(error_message=T('invalid email!'))
IS EXPR Its first argument is a string containing a logical expression in terms of a variable value. It validates a field value if the expression evaluates to True. For example:
1 requires = IS_EXPR('int(value)%3==0', 2 error_message=T('not divisible by 3')) One should first check that the value is an integer so that an exception will not occur.
1 requires = [IS_INT_IN_RANGE(0, 100), IS_EXPR('value%3==0')]
IS FLOAT IN RANGE Checksthatthefield valueis afloating pointnumber within a definite range, 0 ≤ value < 100 in the following example:
1 requires = IS_FLOAT_IN_RANGE(0, 100, 2 error_message=T('too small or too large!'))
IS INT IN RANGE Checks that the field value is an integer number within a definite range, 0 ≤ value < 100 in the following example:
1 requires = IS_INT_IN_RANGE(0, 100, 2 error_message=T('too small or too large!'))
IS IN SET Checks that the field values are in a set:
1 requires = IS_IN_SET(['a', 'b', 'c'], 2 error_message=T('must be a or b or c')) The elements of the set must always be strings unless this validator is pre- ceded by IS INT IN RANGE (which converts the value to int) or IS FLOAT IN RANGE (which converts the value to float). For example:
1 requires = [IS_INT_IN_RANGE(0, 8), IS_IN_SET([2, 3, 5, 7], 2 error_message=T('must be prime and less than 10'))]
IS IN SET and Tagging The IS IN SET validator has an optional attribute multiple=False. If set to True, multiple values can be stored in a field. The field in this case must be a string field. The multiple values are stored separated by a "|". multiple references are handled automatically in create and update forms, but they are transparent to the DAL. We strongly suggest using the jQuery multiselect plugin to render multiple fields. VALIDATORS 205
IS LENGTH Checksif length of field’s valuefits between givenboundaries. Works for both text and file inputs. Its arguments are: • maxsize: the maximum allowed length / size • minsize: the minimum allowed length / size Examples: Check if text string is shorter than 33 characters:
1 INPUT(_type='text', _name='name', requires=IS_LENGTH(32)) Check if password string is longer than 5 characters:
1 INPUT(_type='password', _name='name', requires=IS_LENGTH(minsize=6)) Check if uploaded file has size between 1KB and 1MB:
1 INPUT(_type='file', _name='name', requires=IS_LENGTH(1048576, 1024)) For all field types except for files, it checks the length of the value. In the case of files, the value is a cookie.FieldStorage, so it validates the length of the data in the file, which is the behavior one might intuitively expect.
IS LIST OF This is not properly a validator. Its intended use is to allow validations of fields that return multiple values. It is used in those rare cases when a form contains multiple fields with the same name or a multiple selection box. Its only argumentis anothervalidator, and all it doesis to apply the other validator to each element of the list. For example, the following expression checks that every item in a list is an integer in the range 0-10:
1 requires = IS_LIST_OF(IS_INT_IN_RANGE(0, 10)) It never returns an error and does not contain an error message. The inner validator controls the error generation.
IS LOWER This validator never returns an error. It just converts the value to lower case.
1 requires = IS_LOWER()
IS MATCH This validator matches the value against a regular expression and returns an error if it does not match. Here is an example of usage to validate a US zip code:
1 requires = IS_MATCH('ˆ\d{5}(-\d{4})?$', 2 error_message='not a zip code') Here is an example of usage to validate an IPv4 address:
1 requires = IS_MATCH('ˆ\d{1,3}(\.\d{1,3}){3}$', 2 error_message='not an IP address') 206 FORMS AND VALIDATORS
Here is an example of usage to validate a US phone number:
1 requires = IS_MATCH('ˆ1?((-)\d{3}-?|\(\d{3}\))\d{3}-?\d{4}$', 2 error_message='not a phone number') For more information on Python regular expressions, refer to the official Python documentation.
IS NOT EMPTY This validator checks that the content of the field value is not an empty string.
1 requires = IS_NOT_EMPTY(error_message='cannot be empty!')
IS TIME This validator checks that a field value contains a valid time in the specified format.
1 requires = IS_TIME(error_message=T('must be HH:MM:SS!'))
IS URL Rejects a URL string if any of the following is true: • The string is empty or None • The string uses characters that are not allowed in a URL • The string breaks any of the HTTP syntactic rules • The URL scheme specified (if one is specified) is not ’http’ or ’https’ • The top-level domain (if a host name is specified) does not exist (These rules are based on RFC 2616[61]) Thisfunction onlychecksthe URL’ssyntax. It doesnotcheckthatthe URL points to a real document, for example, or that it otherwise makes semantic sense. This function does automatically prepend ’http://’ in front of a URL in the case of an abbreviated URL (e.g. ’google.ca’). If the parameter mode=’generic’ is used, then this function’s behavior changes. It then rejects a URL string if any of the following is true: • The string is empty or None • The string uses characters that are not allowed in a URL • The URL scheme specified (if one is specified) is not valid (These rules are based on RFC 2396[62]) The list of allowed schemes is customizable with the allowed schemes parameter. If you exclude None from the list, then abbreviated URLs (lacking a scheme such as ’http’) will be rejected. VALIDATORS 207
The default prepended scheme is customizable with the prepend scheme parameter. If you set prepend scheme to None, then prepending will be disabled. URLs that require prepending to parse will still be accepted, but the return value will not be modified. IS URL is compatible with the Internationalized Domain Name (IDN) standard specified in RFC 3490[63]). As a result, URLs can be regular strings or unicode strings. If the URL’s domain component (e.g. google.ca) contains non-US-ASCII letters, then the domain will be converted into Pun- ycode (defined in RFC 3492[64]). IS URL goes a bit beyond the standards, and allows non-US-ASCII characters to be present in the path and query components of the URL as well. These non-US-ASCII characters will be en- coded. For example, space will be encoded as’%20’. The unicode character with hex code 0x4e86 will become ’%4e%86’. Examples:
1 requires = IS_URL()) 2 requires = IS_URL(mode='generic') 3 requires = IS_URL(allowed_schemes=['https']) 4 requires = IS_URL(prepend_scheme='https') 5 requires = IS_URL(mode='generic', allowed_schemes=['ftps', 'https'], prepend_scheme='https')
IS STRONG Enforces complexity requirements on a field (usually a pass- word field) Example:
1 requires = IS_STRONG(min=10, special=2, upper=2) where • min is minimum length of the value • special is the minimum number of required special characters • is the minimum number of upper case characters
IS IMAGE This validator checks if file uploaded through file input was savedin one of selected image formats and has dimensions(width and height) within given limits. It does not check for maximum file size (use IS LENGTH for that). It returns a validation failure if no data was uploaded. It supports the file formats BMP, GIF, JPEG, PNG, and it does not requires the Python Imaging Library. Code parts taken from http://mail.python.org/pipermail/python-list/2007- June/617126.html It takes the following arguments: 208 FORMS AND VALIDATORS
• extensions: iterable containing allowed image file extensions in lower- case (’jpg’ extension of uploaded file counts as ’jpeg’) • maxsize: iterable containing maximum width and height of the image • minsize: iterable containing minimum width and height of the image Use (-1, -1) as minsize to bypass the image-size check. Here are some Examples:
• Check if uploaded file is in any of supported image formats:
1 requires = IS_IMAGE()
• Check if uploaded file is either JPEG or PNG:
1 requires = IS_IMAGE(extensions=('jpeg', 'png'))
• Check if uploaded file is PNG with maximum size of 200x200 pixels:
1 requires = IS_IMAGE(extensions=('png'), maxsize=(200, 200))
IS UPLOAD FILENAME This validator checks if name and extension of file uploaded through file input matches given criteria. It does not ensure the file type in any way. Returns validation failure if no data was uploaded. Its arguments are: • filename: filename (before dot) regex • extension: extension (after dot) regex • lastdot: which dot should be used as a filename / extension separator: True means last dot, e.g., file.png -> file / png False means first dot, e.g., file.tar.gz -> file / tar.gz • case: 0 - keep the case, 1 - transform the string into lowercase(default), 2 - transform the string into uppercase If there is no dot present, extension checks will be done against empty string and filename checks against whole value. Examples: Check if file has a pdf extension (case insensitive):
1 requires = IS_UPLOAD_FILENAME(extension='pdf') Check if file has a tar.gz extension and name starting with backup:
1 requires = IS_UPLOAD_FILENAME(filename='backup.*', extension='tar.gz' , lastdot=False) VALIDATORS 209
Check if file has no extension and name matching README (case sensi- tive):
1 requires = IS_UPLOAD_FILENAME(filename='ˆREADME$', extension='ˆ$', case=0)
IS IPV4 This validator checks if a field’s value is an IP version 4 address in decimal form. Can be set to force addresses from a certain range. IPv4 regex taken from: http://regexlib.com/REDetails.aspx?regexp id=1411 Its arguments are • minip: lowest allowed address; accepts: str, e.g., 192.168.0.1; iterable of numbers, e.g., [192, 168, 0, 1]; int, e.g., 3232235521 • maxip: highest allowed address; same as above All three example values are equal,since addresses are converted to integers for inclusion check with following function:
1 number = 16777216 * IP[0] + 65536 * IP[1] + 256 * IP[2] + IP[3] Examples: Check for valid IPv4 address:
1 requires = IS_IPV4() Check for valid private network IPv4 address:
1 requires = IS_IPV4(minip='192.168.0.1', maxip='192.168.255.255')
IS LOWER This validator never returns an error. It converts the value to lower case.
IS UPPER This validator never returns an error. It converts the value to upper case.
1 requires = IS_UPPER()
IS NULL OR Sometimes you need to allow empty values on a field along with other requirements. For example a field may be a date but it can also be empty. The IS NULL OR validator allows this:
1 requires = IS_NULL_OR(IS_DATE())
CLEANUP This is a filter. It never fails. It just removes all characters whose decimal ASCII codes are not in the list [10, 13, 32-127].
1 requires = CLEANUP() 210 FORMS AND VALIDATORS
CRYPT This is also a filter. It performs a secure hash on the input and it is used to prevent passwords from being passed in the clear to the database.
1 requires = CRYPT(key=None) If the key is None, it uses the MD5 algorithm. If a key is specified it uses the HMAC+SHA512with the providedkey. The keyhas to be a unique string associated to the database used. The key can never be changed. If you lose the key the previously hashed values become useless.
Database Validators
IS NOT IN DB Consider the following example:
1 db.define_table('person', Field('name')) 2 db.person.name.requires = IS_NOT_IN_DB(db, 'person.name') It requires that when you insert a new person, his/her name is not already in the database, db, in the field person.name. As with all other validators this requirement is enforced at the form processing level, not at the database level. This means that there is a small probability that, if two visitors try to concurrently insert records with the same person.name, this results in a race condition and both records are accepted. It is therefore safer to also inform the database that this field should have a unique value:
1 db.define_table('person', Field('name', unique=True)) 2 db.person.name.requires = IS_NOT_IN_DB(db, 'person.name') Now if a race condition occurs, the database raises an OperationalError and one of the two inserts is rejected. The first argument of IS NOT IN DB can be a database connection or a DAL SSet. In the latter case, you would be checking only the set defined by the Set. The following code, for example, does not allow registration of two persons with the same name within 10 days of each other:
1 import datetime 2 now = datetime.datetime.today() 3 db.define_table('person', 4 Field('name'), 5 Field('registration_stamp', 'datetime', default=now)) 6 recent = db(db.person.registration_stamp>now-datetime.timedelta(10)) 7 db.person.name.requires = IS_NOT_IN_DB(recent, 'person.name')
IS IN DB Consider the following tables and requirement:
1 db.define_table('person', Field('name', unique=True)) 2 db.define_table('dog', Field('name'), Field('owner', db.person) 3 db.dog.owner.requires = IS_IN_DB(db, 'person.id', '%(name)s') VALIDATORS 211
It is enforced at the level of dog INSERT/UPDATE/DELETE forms. It requires that a dog.owner be a valid id in the field person.id in the database db. Because of this validator, the dog.owner field is represented as a dropbox. The third argument of the validator is a string that describes the elements in the dropbox. In the example you want to see the person %(name)s instead of the person %(id)s. %(...)s is replaced by the value of the field in brackets for each record. If you want the field validated, but you do not want a dropbox, you must put the validator in a list.
1 db.dog.owner.requires = [IS_IN_DB(db, 'person.id', '%(name)s')] The first argument of the validator can be a database connection or a DAL Set, as in IS NOT IN DB.
IS IN DB and Tagging The IS IN DB validator has an optional attribute multiple=False. If set to true multiple values can be stored in a field. The field in this case cannot be a reference but it must be a string field. The multiple values are stored separated by a "|". multiple references are handled automatically in create and update forms, but they are transparent to the DAL. We strongly suggest using the jQuery multiselect plugin to render multiple fields.
Custom Validators All validators follow the prototype below:
1 class sample_validator: 2 def __init__(self, *a, error_message='error'): 3 self.a = a 4 self.e = error_message 5 def __call__(value): 6 if validate(value): 7 return (parsed(value), None) 8 return (value, self.e) 9 def formatter(self, value): 10 return format(value)
i.e., when called to validate a value, a validator returns a tuple (x, y). If y is None, then the value passed validation and x contains a parsed value. For example, if the validator requires the value to be an integer, x is converted to int(value). If the value did not pass validation, then x contains the input value and y contains an error message that explains the failed validation. This error message is used to report the error in forms that do not validate. The validator may also contain a formatter method. It must perform the opposite conversion to the one the call does. For example, consider the source code for IS DATE: 212 FORMS AND VALIDATORS
1 class IS_DATE(object): 2 def __init__(self, format='%Y-%m-%d', error_message='must be YYYY -MM-DD!'): 3 self.format = format 4 self.error_message = error_message 5 def __call__(self, value): 6 try: 7 y, m, d, hh, mm, ss, t0, t1, t2 = time.strptime(value, str(self.format)) 8 value = datetime.date(y, m, d) 9 return (value, None) 10 except: 11 return (value, self.error_message) 12 def formatter(self, value): 13 return value.strftime(str(self.format))
On success, the call method reads a date string from the form and converts it into a datetime.date object using the format string specified in the constructor. The formatter object takes a datetime.date object and converts it to a string representation using the same format. The formatter is called automatically in forms, but you can also call it explicitly to convert objects into their proper representation. For example:
1 >>> db = DAL() 2 >>> db.define_table('atable', 3 Field('birth', 'date', requires=IS_DATE('%m/%d/%Y'))) 4 >>> id = db.atable.insert(birth=datetime.date(2008, 1, 1)) 5 >>> rows = db(db.atable.id==id).select() 6 >>> print db.atable.formatter(rows[0].birth) 7 01/01/2008 When multiple validators are required (and stored in a list), they are exe- cuted in order and the output of one is passed as input to the next. The chain breaks when one of the validators fails. Conversely, when we call the formatter method of a field, the formatters of the associated validators are also chained, but in reverse order.
Validators with Dependencies Occasionally, you need to validate a field and the validator depends on the value of another field. This can be done, but it requires setting the validator in the controller, when the value of the other field is known. For example, here is a page that generates a registration form that asks for username and password twice. None of the fields can be empty, and both passwords must match:
1 def index(): 2 match_it = IS_EXPR('value==%s' % repr(request.vars.password), 3 error_message='passwords do not match') 4 form = SQLFORM.factory( WIDGETS 213
5 Field('username', requires=IS_NOT_EMPTY()), 6 Field('password', requires=IS_NOT_EMPTY()), 7 Field('password_again', requires=match_it)) 8 if form.accepts(request.vars, session): 9 pass # or take some action 10 return dict(form=form) The same mechanism can be applied to FORM and SQLFORM objects.
7.5 Widgets
Here is a list of available web2py widgets:
1 SQLFORM.widgets.string.widget 2 SQLFORM.widgets.text.widget 3 SQLFORM.widgets.password.widget 4 SQLFORM.widgets.integer.widget 5 SQLFORM.widgets.double.widget 6 SQLFORM.widgets.time.widget 7 SQLFORM.widgets.date.widget 8 SQLFORM.widgets.datetime.widget 9 SQLFORM.widgets.upload.widget 10 SQLFORM.widgets.boolean.widget 11 SQLFORM.widgets.options.widget 12 SQLFORM.widgets.multiple.widget 13 SQLFORM.widgets.radio.widget 14 SQLFORM.widgets.checkboxes.widget The first ten of them are the defaults for the corresponding field types. The "options" widget is used when a field’s requires is IS IN SETS or I IN DB with multiple=False (default behavior). The "multiple" widget is used when a field’s requires is IS IN SETS or I IN DB with multiple=True. The "radio" and "checkboxes" widgets are never used by default, but can be set manually. For example, to have a "string" field represented by a textarea:
1 Field('comment', 'string', widget=SQLFORM.widgets.text.widget) You can create new widgets or extend existing widgets; in fact, SQLFORM.widgets[type] isaclassand SQLFORM.widgets[type].widget is a static member function of the corresponding class. Each widget function takes two arguments: the field object, and the current value of that field. It returns a representation of the widget. As an example, the string widget could be recoded as follows:
1 def my_string_widget(field, value): 2 return INPUT(_name=field.name, 3 _id="%s_%s\" % (field._tablename, field.name), 4 _class=field.type, 5 _value=value, 6 requires=field.requires) 214 FORMS AND VALIDATORS
7 8 Field('comment', 'string', widget=my_string_widget) The id and class values must follow the convention described later in this chapter. A widget may contain its own validators, but it is good practice to associate the validators to the "requires" attribute of the field and have the widget get them from there.
7.6 CRUD
One of the recent additions to web2py is the Create/Read/Update/Delete (CRUD) API on top of SQLFORM. CRUD creates an SQLFORM, but it simplifies the coding because it incorporates the creation of the form, the processing of the form, the notification, and the redirection, all in one single function. What that function is depends on what you want to do. The first thing to notice is that CRUD differs from the other web2py APIs we have used so far because it is not already exposed. It must be imported. It also must be linked to a specific database. For example:
1 from gluon.tools import Crud 2 crud = Crud(globals(), db)
The first argument of the constructor is the current context globals() so that CRUD can access the local request, response, and session. The second argument is a database connection object, db. The crud object defined above provides the following API: .
• crud.tables() returns a list of tables defined in the database.
• crud.create(db.tablename) returns a create form for table tablename.
• crud.read(db.tablename, id) returns a readonly form for tablename and record id.
• crud.update(db.tablename, id) returns an update form for tablename and record id.
• crud.delete(db.tablename, id) deletes the record.
• crud.select(db.tablename, query) returns a list of records selected from the table.
• crud() returns one of the above based on the request.args(). For example, the following action: CRUD 215
1 def data: return dict(form=crud()) would expose the following URLs:
1 http://.../[app]/[controller]/data/tables 2 http://.../[app]/[controller]/data/create/[tablename] 3 http://.../[app]/[controller]/data/read/[tablename]/[id] 4 http://.../[app]/[controller]/data/delete/[tablename] 5 http://.../[app]/[controller]/data/select/[tablename] However, the following action:
1 def create_tablename: 2 return dict(form=crud.create(db.tablename)) would only expose the create method
1 http://.../[app]/[controller]/create_tablename While the following action:
1 def update_tablename: 2 return dict(form=crud.update(db.tablename, request.args(0))) would only expose the update method
1 http://.../[app]/[controller]/update_tablename and so on. The behavior of CRUD can be customized in two ways: by setting some attributes of the crud object or by passing extra parameters to each of its methods.
Attributes Here is a complete list of current CRUD attributes, their default values, and meaning:
1 crud.settings.create_next = request.url specifies the URL to redirect to after a successful "create" record.
1 crud.settings.update_next = request.url specifies the URL to redirect to after a successful "update" record.
1 crud.settings.delete_next = request.url specifies the URL to redirect to after a successful "delete" record.
1 crud.settings.download_url = URL(r=request, f='download') specifies the URL to be used for linking uploaded files.
1 crud.settings.create_onvalidation = lambda form: None 216 FORMS AND VALIDATORS
is an optional function to be called onvalidation of "create" forms (see SQL- FORM onvalidation)
1 crud.settings.update_onvalidation = lambda form: None
is an optional function to be called onvalidation of "update" forms (see SQLFORM onvalidation)
1 crud.settings.create_onaccept = lambda form: None is an optional function to be called before redirect after successful "create" record. This function takes the form as its only argument.
1 crud.settings.update_onaccept = lambda form: None is an optional function to be called before redirect after successful "update" record. This function takes the form as its only argument.
1 crud.settings.update_ondelete = lambda form: None is an optional function to be called before redirect after successfully deleting a record using an "update" form. This function takes the form as its only argument.
1 crud.settings.delete_onaccept = lambda record: None is an optional function to be called before redirect after successfully deleting a record using the "delete" method. This function takes the form as its only argument.
1 crud.settings.update_deletable = True determines whether the "update" forms should have a "delete" button.
1 crud.settings.showid = False determines whether the "update" forms should show the id of the edited record.
1 crud.settings.keepvalues = False determines whether forms should keep the previously inserted values or reset to default after successful submission.
Messages Here is a list of customizable messages:
1 crud.messages.submit_button = 'Submit' sets the text of the "submit" button for both create and update forms.
1 crud.messages.delete_label = 'Check to delete:' sets the label of the "delete" button in "update" forms. CRUD 217
1 crud.messages.record_created = 'Record Created' sets the flash message on successful record creation.
1 crud.messages.record_updated = 'Record Updated' sets the flash message on successful record update.
1 crud.messages.record_deleted = 'Record Deleted' sets the flash message on successful record deletion.
1 crud.messages.update_log = 'Record %(id)s updated' sets the log message on successful record update.
1 crud.messages.create_log = 'Record %(id)s created' sets the log message on successful record creation.
1 crud.messages.read_log = 'Record %(id)s read' sets the log message on successful record read access.
1 crud.messages.delete_log = 'Record %(id)s deleted' sets the log message on successful record deletion. Notice that crud.messages belongs to the class gluon.storage.Message which is similar to gluon.storage.Storage but it automatically translates its values, without need for the T operator. Log messages are used if and only if CRUD is connected to Auth as discussedin Chapter 8. The eventsare loggedin the Auth table "auth events".
Methods The behavior of CRUD methods can also be customized on a per call basis. Here are their signatures:
1 crud.tables() 2 crud.create(table, next, onvalidation, onaccept, log, message) 3 crud.read(table, record) 4 crud.update(table, record, next, onvalidation, onaccept, ondelete, log, message, deletable) 5 crud.delete(table, record_id, next, message) 6 crud.select(table, query, fields, orderby, limitby, headers, **attr)
• table isaDALtableoratablenamethemethodshouldacton.
• record and record id are the id of the record the method should act on.
• next is the URL to redirect to after success. If the URL contains the substring "[id]" this will be replaced by the id of the record currently created/updated. 218 FORMS AND VALIDATORS
• onvalidation has the same function as SQLFORM(..., onvalidation)
• onaccept is a function to be called after the form submission is accepted and acted upon, but before redirection.
• log is the log message. Log messages in CRUD see variables in the form.vars dictionary such as "%(id)s".
• message is the flash message upon form acceptance.
• ondelete is called in place of onaccept when a record is deleted via an "update" form.
• deletable determines whether the "update" form should have a delete option.
• query is the query to be used to select records.
• fields is a list of fields to be selected.
• orderby determines the order in which records should be selected (see Chapter 6).
• limitby determines the range of selected records that should be dis- played (see Chapter 6).
• headers is a dictionary with the table header names.
Here is an example of usage in a single controller function:
1 # assuming db.define_table('person', Field('name')) 2 def people(): 3 form = crud.create(db.person, next=request.url, 4 message=T("record created")) 5 persons = crud.select(db.person, fields=['name'], 6 headers={'person.name', 'Name'}) 7 return dict(form=form, persons=persons)
7.7 Custom form
If a form is created with SQLFORM, SQLFORM.factory or CRUD, there are multiple ways it can be embedded in a view allowing multiple degrees of customization. Consider for example the following model:
1 db.define_table('image', 2 Field('name'), 3 Field('file', 'upload')) CUSTOM FORM 219
and upload action
1 def upload_image(): 2 return dict(form=crud.create(db.image)) The simplest way to embed the form in the view for upload image is
1 {{=form}} This results in a standardtable layout. If you wish to use a different layout, you can break the form into components
1 {{=form.custom.begin}} 2 Image name:
If you do not wish to use the widgets serialized by web2py, you can replace them with HTML. There are some variables that will be useful for this: • form.custom.labels[fieldname] contains the label for the field.
• form.custom.dspval[fieldname] form-type and field-type dependentdis- play representation of the field.
• form.custom.inpval[fieldname] form-type and field-type dependentval- ues to be used in field code. It is important to follow the conventions described below. 220 FORMS AND VALIDATORS
CSS Conventions Tags in forms generated by SQLFORM, SQLFORM.factory and CRUD fol- low a strict CSS naming convention that can be used to further customize the forms. Given a table "mytable", a field "myfield" of type "string", it is rendered by default by a
1 SQLFORM.widgets.string.widget that looks like this:
1 Notice that: • the class of the INPUT tag is the same as the type of the field. This is very important for the jQuery code in "web2py ajax.html" to work. It makes sure that you can only have numbers in "integer" and "double" fields, and that "time", "date" and "datetime" fields display the popup calendar. • the id is the name of the class plus the name of the field, joined by one underscore. This allows you to uniquely refer to the field via jQuery(’#mytable myfield’) and manipulate,forexample,the stylesheet of the field or bind actions associated to the field events (focus, blur, keyup, etc.). • the name is, as you would expect, the field name.
Switch off errors Occasionally, you may want to disable the automatic error placement and display form error messages in some place other than the default. That can be done in two steps: • display the error messages where desired
• form.error.clear() before the form is rendered so that error messages are not displayed in the default locations. Here is an example where the errors are displayed above the form and not in the form.
1 {{if form.errors:}} 2 Your submitted form contains the following errors: CUSTOM FORM 221
3
- 4 {{for fieldname in form.errors:}} 5
CHAPTER 8
ACCESS CONTROL
web2py includes a powerful and customizable Role-Based Access Control (RBAC) mechanism. Here is a definition from Wikipedia: “Role-Based Access Control (RBAC) is an approach to restricting system access to authorized users. It is a newer alternative approach to mandatory access control (MAC) and discretionary access control (DAC). RBAC is sometimes referred to as role-based security. RBAC is a policyneutral andflexible accesscontrol technologysufficiently powerful to simulate DAC and MAC. Conversely, MAC can simulate RBAC if the role graph is restricted to a tree rather than a partially ordered set. Prior to the development of RBAC, MAC and DAC were considered to be the only known models for access control: if a model was not MAC, it was considered to be a DAC model, and vice versa. Research in the late 1990s demonstrated that RBAC falls in neither category. Within an organization, roles are created for various job functions. The permissions to perform certain operations are assigned to specific roles. Mem- bers of staff (or other system users) are assigned particular roles, and through
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 223 Copyright © 2009 224 ACCESS CONTROL those role assignments acquire the permissions to perform particular system functions. Unlike context-basedaccesscontrol (CBAC), RBAC does not look at the message context (such as a connection’s source). Since users are not assigned permissions directly, but only acquire them through their role (or roles), management of individual user rights becomes a matter of simply assigning appropriate roles to the user; this simplifies common operations, such as adding a user, or changing a user’s department. RBAC differs from access control lists (ACLs) used in traditional dis- cretionary access control systems in that it assigns permissions to specific operations with meaning in the organization, rather than to low level data objects. For example, an access control list could be used to grant or deny write access to a particular system file, but it would not dictate how that file could be changed.” The web2py class that implements RBAC is called Auth. Auth needs (and defines) the following tables: • auth user stores users’ name, email address, password, and status (reg- istration pending, accepted, blocked)
• auth group stores groups or roles for users in a many-to-many structure. By default, each user is in its own group, but a user can be in multiple groups,andeachgroupcancontainmultipleusers. Agroupisidentified by a role and a description.
• auth membership links users and groups in a many-to-many structure.
• auth permission links groups and permissions. A permission is iden- tified by a name and, optionally, a table and a record. For example, members of a certain group can have "update" permissionson a specific record of a specific table.
• auth event logs changes in the other tables and successful access via CRUD to objects controlled by the RBAC. In principle, there is no restriction on the names of the roles and the names of the permissions; the developer can create them to fix the roles and permissions in the organization. Once they have been created, web2py providesanAPIto checkif auseris loggedin, ifa useris amemberofa given group, and/or if the user is a member of any group that has a given required permission. web2py also provides decorators to restrict access to any function based on login, membership and permissions. web2py also understands some specific permissions, i.e., those that have a name that correspond to the CRUD methods (create, read, update, delete) and can enforce them automatically without the need to use decorators. AUTHENTICATION 225
In this chapter, we are going to discuss different parts of RBAC one by one.
8.1 Authentication
In order to use RBAC, users need to be identified. This means that they need to register (or be registered) and log in. Auth provides multiple login methods. The default one consists of iden- tifying users based on the local auth user table. Alternatively, it can log in users against third-party basic authentication systems (for example a Twit- ter account), SMTP servers (for example Gmail), or LDAP (your corporate account). It can also use third-party single-sign-on systems, for example Google. This is achieved via plugins, and new plugins are added all the time. To start using Auth, you need at least this code in a model file, which is also provided with the web2py "welcome" application and assumes a db connection object:
1 from gluon.tools import Auth 2 auth = Auth(globals(), db) 3 auth.define_tables() To expose Auth, you also need the following function in a controller (for example in "default.py"):
1 def user(): return dict(form=auth())
The auth objectandthe user action are already defined in the scaffolding application.
web2py also includes a sample view "default/user.html" to render this function properly that looks like this:
1 {{extend 'layout.html'}} 2
{{=request.args(0)}}
3 {{=form}} 4 {{if request.args(0)=='login':}} 5 register6 lost password
7 {{pass}} The controller above exposes multiple actions:
1 http://.../[app]/default/user/register 2 http://.../[app]/default/user/login 3 http://.../[app]/default/user/logout 4 http://.../[app]/default/user/profile 5 http://.../[app]/default/user/change_password 226 ACCESS CONTROL
6 http://.../[app]/default/user/verify_email 7 http://.../[app]/default/user/retrieve_username 8 http://.../[app]/default/user/retrieve_password 9 http://.../[app]/default/user/impersonate 10 http://.../[app]/default/user/groups 11 http://.../[app]/default/user/not_authorized
• register allows users to register. It is integrated with CAPTCHA, although this is disabled by default. • login allows users who are registered to log in (if the registration is verified or does not require verification, if it has been approved or does not require approval, and if it has not been blocked). • logout does what you would expect but also, as the other methods, logs the event and can be used to trigger some event.
• profile allows usersto edit their profile, i.e. the contentof the auth user table. Notice that this table does not have a fixed structure and can be customized. • change password allows users to change their password in a fail-safe way. • verify email. If email verification is turned on, then visitors, upon reg- istration, receive an email with a link to verify their email information. The link points to this action. • retrieve username.By default, Auth uses email and password for login, but it can, optionally, use username instead of email. In this latter case, if a user forgets his/her username, the retrieve username method allows the user to type the email address and retrieve the username by email. • retrieve password. Allows users who forgot their password to receive a new one by email. The name here can be misleading because this function does not retrieve the current password (that would be impos- sible since the password is only stored encrypted/hashed) but generates a new one. • impersonate allows a user to "impersonate" another user. This is important for debugging and for support purposes. request.args[0] is the id of the user to be impersonated. This is only allowed if the logged in user has permission(’impersonate’, db.auth user, user id). • groups lists the groups the current logged in user is a member of. AUTHENTICATION 227
• not authorized displays an error message when the visitor tried to do something that he/she is not authorized to do. Logout, profile, change password, impersonate, and groups require login. By default they are all exposed, but it is possible to restrict access to only some of these actions. All of the methods abovecanbe extendedor replaced by subclassing Auth. To restrict access to functions to only logged in visitors, decorate the function as in the following example
1 @auth.requires_login() 2 def hello(): 3 return dict(message='hello logged in visitor') Any function can be decorated, not just exposed actions. Of course this is still only a very simple example of access control. More complex examples will be discussed later.
Email verification By default, email verification is disabled. To enable email, append the fol- lowing lines in the model where auth is defined:
1 from gluon.tools import Mail 2 mail = Mail(globals()) 3 mail.settings.server = 'smtp.example.com:25' 4 mail.settings.sender = 'you@example.com' 5 mail.settings.login = 'username:password' 6 auth.settings.mailer = mail 7 auth.settings.registration_requires_verification = False 8 auth.messages.verify_email_subject = 'Email verification' 9 auth.messages.verify_email = \ 10 'Click on the link http://...verify_email/%(key)s to verify your email' You need to replace the mail.settings with the proper parameters for your SMTP server. Set mail.settings.login=False if the SMTP server does not require authentication. You also need to replace the string
1 'Click on the link ...'
in auth.messages.verify email with the proper complete URL of the action verify email. This is necessary because web2py may be installed behind a proxy, and it cannot determine its own public URLs with absolute certainty. Once mail is defined, it can also be used to send email explicitly via
1 mail.send(to=['somebody@example.com'], 2 subject='hello', message='hi there') 228 ACCESS CONTROL
Restrictions on registration If you want to allow visitors to register but not to log in until registration has been approved by the administrator:
1 auth.settings.registration_requires_approval = True You can approve a registration via the appadmin interface. Look into the table auth user. Pending registrations have a registration key field set to "pending". A registration is approved when this field is set to blank. Via the appadmin interface, you can also block a user from logging in. Lo- cate the user in the table auth user and set the registration key to "blocked". "blocked" users are not allowed to log in. Notice that this will prevent a visitor from logging in but it will not force a visitor who is already logged in to log out. You can also block access to the "register" page completely with this statement:
1 auth.settings.actions_disabled.append('register') Other methods of Auth can be restricted in the same way.
CAPTCHA and reCAPTCHA To prevent spammers and bots registering on your site, you may require a registration CAPTCHA. web2py supports reCAPTCHA [65] out of the box. This is because reCAPTCHA is very well designed, free, accessible (it can read the words to the visitors), easy to set up, and does not require installing any third-party libraries. This is what you need to do to use reCAPTCHA: • Register with reCAPTCHA [65] and obtain a (PUBLIC KEY, PRI- VATE KEY) couple for your account. These are just two strings.
• Append the following code to your model after the auth object is defined:
1 from gluon.tools import Recaptcha 2 auth.settings.captcha = Recaptcha(request, 3 'PUBLIC_KEY', 'PRIVATE_KEY')
reCAPTCHA may not work if you access the web site as ’localhost’ or ’127.0.0.1’, because it is registered to work with publicly visible web sites only. The Recaptcha constructor takes some optional arguments:
1 Recaptcha(..., use_ssl=True, error_message='invalid') AUTHENTICATION 229
Notice that use ssl=False by default. If you do not want to use reCAPTCHA, look into the definition of the Recaptcha class in "gluon/tools.py", since it is easy to use other CAPTCHA systems.
Customizing Auth The call to
1 auth.define_tables() defines all Auth tables that have not been defined already. This means that if you wish to do so, you can define your own auth user table. Using a similar syntax to the one show below, you can customize any other Auth table. Here is the proper way to define a user table:
1 # after 2 # auth = Auth(globals(),db) 3 4 auth_table = db.define_table( 5 auth.settings.table_user_name, 6 Field('first_name', length=128, default=''), 7 Field('last_name', length=128, default=''), 8 Field('email', length=128, default='', unique=True), 9 Field('password', 'password', length=256, 10 readable=False, label='Password'), 11 Field('registration_key', length=128, default= '', 12 writable=False, readable=False)) 13 14 auth_table.first_name.requires = \ 15 IS_NOT_EMPTY(error_message=auth.messages.is_empty) 16 auth_table.last_name.requires = \ 17 IS_NOT_EMPTY(error_message=auth.messages.is_empty) 18 auth_table.password.requires = [IS_STRONG(), CRYPT()] 19 auth_table.email.requires = [ 20 IS_EMAIL(error_message=auth.messages.invalid_email), 21 IS_NOT_IN_DB(db, auth_table.email)] 22 auth.settings.table_user = auth_table 23 24 # before 25 # auth.define_tables() You can add any field you wish, but you cannot remove the required fields shown in this example. It is important to make "password" and "registration key" fields readable=False and make the "registration key" field writable=False, since a visitor must not be allowed to tamper with them. If you add a field called "username", it will be used in place of "email" for login. If you do, you will need to add a validator as well:
1 auth_table.username.requires = IS_NOT_IN_DB(db, auth_table.username) 230 ACCESS CONTROL
Renaming Auth tables
The actual names of the Auth tables are stored in
1 auth.settings.table_user_name = 'auth_user' 2 auth.settings.table_group_name = 'auth_group' 3 auth.settings.table_membership_name = 'auth_membership' 4 auth.settings.table_permission_name = 'auth_permission' 5 auth.settings.table_event_name = 'auth_event' The names of the table can be changed by reassigning the above variables after the auth object is defined and before the Auth tables are defined. For example:
1 auth = Auth(globals(),db) 2 auth.settings.table_user_name = 'person' 3 #... 4 auth.define_tables() The actual tables can also be referenced, independently of their actual names, by
1 auth.settings.table_user 2 auth.settings.table_group 3 auth.settings.table_membership 4 auth.settings.table_permission 5 auth.settings.table_event
Alternate Login Methods Auth provides multiple login methods and hooks to create new login methods. Each supported login method corresponds to a file in the folder
1 gluon/contrib/login_methods/ Refer to the documentation in the files themselves for each login method, but here we provide some examples. First of all we need to make a distinction between two types of alternate login methods:
• login methods that use a web2py form (although the credentials are verified outside web2py). An example is LDAP.
• login methods that require an external sign-on (web2py never gets to see the credentials).
Let’s consider examples of the first case: AUTHENTICATION 231
Basic Let’s say you have an authentication service, for example at the url https://basic.example.com, that accepts basic access authentication. That means the server accepts HTTP requests with a header of the form:
1 GET /index.html HTTP/1.0 2 Host: basic.example.com 3 Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== where the latter string is the base64 encoding of the string username:password. The service responds 200 OK if the user is authorized and 400, 401, 402, 403 or 404 otherwise. You want to enter username and password using the standard Auth login form and verify the credentials against such a service. All you need to do is add the following code to your application
1 from gluon.contrib.login_methods.basic_auth import basic_auth 2 auth.settings.login_methods.append( 3 basic_auth('https://basic.example.com'))
Notice that auth.settings.login methods is a list of authentication methods that are executed sequentially. By default it is set to
1 auth.settings.login_methods = [auth]
When an alternate method is appended, for example basic auth, Auth first tries to log in the visitor based on the content of auth user, and when this fails, it tries the next method in the list. If a method succeeds in logging in the visitor, and if auth.settings.login methods[0]==auth, Auth takes the following actions:
• if the user does not exist in auth user, a new user is created and the username/email and passwords are stored.
• if the user does exist in auth user but the new accepted password does not match the old stored password, the old password is replaced with the new one (notice that passwords are always stored hashed unless specified otherwise).
If you do not wish to store the new password in auth user, then it is sufficient to change the order of login methods, or remove auth from the list. For example:
1 from gluon.contrib.login_methods.basic_auth import basic_auth 2 auth.settings.login_methods = \ 3 [basic_auth('https://basic.example.com')] The same applies for any other login method described here. 232 ACCESS CONTROL
SMTP and Gmail You can verify the login credentials using a remote SMTP server, for example Gmail; i.e., you log the user in if the email and password they provide are valid credentials to access the Gmail SMTP server (smtp.gmail.com:587). All that is needed is the following code:
1 from gluon.contrib.login_methods.email_auth import email_auth 2 auth.settings.login_methods.append( 3 email_auth("smtp.gmail.com:587", "@gmail.com"))
The first argument of email auth is the address:port of the SMTP server. The second argument is the email domain. This works with any SMTP server that requires TLS authentication.
LDAP Authentication using LDAP works very much as in the previous cases. To use LDAP login with MS Active Directory:
1 from gluon.contrib.login_methods.ldap_auth import ldap_auth 2 auth.settings.login_methods.append(ldap_auth(mode='ad', 3 server='my.domain.controller', 4 base_dn='ou=Users,dc=domain,dc=com'))
To use LDAP login with Lotus Notes and Domino:
1 auth.settings.login_methods.append(ldap_auth(mode='domino', 2 server='my.domino.server'))
To use LDAP login with OpenLDAP (with UID):
1 auth.settings.login_methods.append(ldap_auth(server='my.ldap.server', 2 base_dn='ou=Users,dc=domain,dc=com'))
To use LDAP login with OpenLDAP (with CN):
1 auth.settings.login_methods.append(ldap_auth(mode='cn', 2 server='my.ldap.server', base_dn='ou=Users,dc=domain,dc=com'))
Google on GAE Authentication using Google when running on Google App Enginerequires skipping the web2py login form, being redirected to the Google login page, and back upon success. Because the behavior is different than in the previous examples, the API is a little different.
1 from gluon.contrib.login_methods.gae_google_login import GaeGoogleAccount 2 auth.settings.login_form = GaeGoogleAccount() AUTHORIZATION 233
8.2 Authorization
Once a new user is registered, a new group is created to contain the user. The role of the new user is conventionally "user [id]" where [id] is the id of the newly created id. The creation of the group can be disabled with
1 auth.settings.create_user_groups = False although we do not suggest doing so. Usershavemembershipin groups. Eachgroup is identifiedby a name/role. Groups have permissions. Users have permissions because of the groups they belong to. You can create groups, give membership and permissions via appadmin or programmatically using the following methods:
1 auth.add_group('role', 'description') returns the id of the newly created group.
1 auth.del_group(group_id)
deletes the group with group id.
1 auth.del_group(auth.id_group('user_7')) deletes the group with role "user 7", i.e., the group uniquely associated to user number 7.
1 auth.user_group(user_id)
returnstheid ofthegroupuniquelyassociatedto theuseridentified by user id.
1 auth.add_membership(group_id, user_id)
gives user id membership of the group group id. If the user id is notspecified, then web2py assumes the current logged-in user.
1 auth.del_membership(group_id, user_id)
revokes user id membership of the group group id. If the user id is not specified, then web2py assumes the current logged-in user.
1 auth.has_membership(group_id, user_id)
checks whether user id has membership of the group group id. If the user id is not specified, then web2py assumes the current logged-in user.
1 auth.add_permission(group_id, 'name', 'object', record_id) gives permission "name" (user defined) on the object "object" (also user defined) to members of the group group id. If "object" is a tablename then the permission can refer to the entire table (record id==0) or to a specific record (record id>0). When giving permissions on tables, it is common to use a permission name in the set (’create’, ’read’, ’update’, ’delete’, ’select’) since these permissions are understood and can be enforced by CRUD. 234 ACCESS CONTROL
1 auth.del_permission(group_id, 'name', 'object', record_id) revokes the permission.
1 auth.has_permission('name', 'object', record_id, user_id)
checks whether the user identified by user id has membership in a group with the requested permission.
1 rows = db(accessible_query('read', db.sometable, user_id))\ 2 .select(db.mytable.ALL)
returns all rows of table "sometable" that user user id has "read" permission on. If the user id is not specified, then web2py assumes the current logged- in user. The accessible query(...) can be combined with other queries to make more complex ones. accessible query(...) is the only Auth method to require a JOIN, so it does not work on the Google App Engine. Assuming the following definitions:
1 >>> from gluon.tools import Auth 2 >>> auth = Auth(globals(), db) 3 >>> auth.define_tables() 4 >>> secrets = db.define_table('document', Field('body')) 5 >>> james_bond = db.auth_user.insert(first_name='James', 6 last_name='Bond') Here is an example:
1 >>> doc_id = db.document.insert(body = 'top secret') 2 >>> agents = auth.add_group(role = 'Secret Agent') 3 >>> auth.add_membership(agents, james_bond) 4 >>> auth.add_permission(agents, 'read', secrets) 5 >>> print auth.has_permission('read', secrets, doc_id, james_bond) 6 True 7 >>> print auth.has_permission('update', secrets, doc_id, james_bond) 8 False
Decorators The most common way to check permission is not by explicit calls to the above methods, but by decorating functions so that permissions are checked relative to the logged-in visitor. Here are some examples:
1 def function_one(): 2 return 'this is a public function' 3 4 @auth.requires_login() 5 def function_two(): 6 return 'this requires login' 7 8 @auth.requires_membership('agents') 9 def function_three(): AUTHORIZATION 235
10 return 'you are a secret agent' 11 12 @auth.requires_permission('read', secrets) 13 def function_four(): 14 return 'you can read secret documents' 15 16 @auth.requires_permission('delete', 'any file') 17 def function_five(): 18 import os 19 for file in os.listdir('./'): 20 os.unlink(file) 21 return 'all files deleted' 22 23 @auth.requires_permission('add', 'number') 24 def add(a, b): 25 return a + b 26 27 def function_six(): 28 return add(3, 4) Note that access to all functions apart from the first one is restricted based on permissions that the visitor may or may not have. If the visitor is not logged in, then the permission cannot be checked; the visitor is redirected to the login page and then back to the page that requires permissions. If the visitor doesnot havepermissionto accessa givenfunction, the visitor is redirect to the URL defined by
1 auth.settings.on_failed_authorization = \ 2 URL (r=request, f='user/on_failed_authorization') You can change this variable and redirect the user elsewhere.
Combining requirements Occasionally, it is necessary to combine requirements. This can be done via a generic requires decorator which takes a single argument, a true or false condition. For example, to give access to agents, but only on Tuesday:
1 @auth.requires(auth.has_membership(agents) \ 2 and request.now.weekday()==1) 3 def function_seven(): 4 return 'Hello agent, it must be Tuesday!'
Authorization and CRUD Using decorators and/or explicit checks provides one way to implement access control. 236 ACCESS CONTROL
Another way to implement access control is to always use CRUD (as opposed to SQLFORM) to access the database and to ask CRUD to enforce access control on database tables and records. This is done by linking Auth and CRUD with the following statement:
1 crud.settings.auth = auth This will prevent the visitor from accessing any of the CRUD functions unless the visitor is logged in and has explicit access. For example, to allow a visitor to post comments, but only update their own comments (assuming crud, auth and db.comment are defined):
1 def give_create_permission(form): 2 group_id = auth.id_group('user_%s' % auth.user.id) 3 auth.add_permission(group_id, 'read', db.comment) 4 auth.add_permission(group_id, 'create', db.comment) 5 auth.add_permission(group_id, 'select', db.comment) 6 7 def give_update_permission(form): 8 comment_id = form.vars.id 9 group_id = auth.id_group('user_%s' % auth.user.id) 10 auth.add_permission(group_id, 'update', db.comment, comment_id) 11 auth.add_permission(group_id, 'delete', db.comment, comment_id) 12 13 auth.settings.register_onaccept = give_create_permission 14 crud.settings.auth = auth 15 16 def post_comment(): 17 form = crud.create(db.comment, onaccept=give_update_permission) 18 comments = db(db.comment.id>0).select() 19 return dict(form=form, comments=comments) 20 21 def update_comment(): 22 form = crud.update(db.comment, request.args(0)) 23 return dict(form=form) You can also select specific records (those you have ’read’ access to):
1 def post_comment(): 2 form = crud.create(db.comment, onaccept=give_update_permission) 3 query = auth.accessible_query('read', db.comment, auth.user.id) 4 comments = db(query).select(db.comment.ALL) 5 return dict(form=form, comments=comments)
Authorization and Downloads
The use of decorators and the use of crud.settings.auth do not enforce authorization on files downloaded by the usual download function
1 def download(): return response.download(request, db) If one wishes to do so, one must declare explicitly which "upload" fields contain files that need access control upon download. For example: AUTHORIZATION 237
1 db.define_table('dog', 2 Field('small_image', 'upload') 3 Field('large_image', 'upload')) 4 5 db.dog.large_image.authorization = lambda record: \ 6 auth.is_logged_in() and \ 7 auth.has_permission('read', db.dog, record.id, auth.user.id)
The attribute authorization of upload field can be None (the default) or a function thatdecides whetherthe useris logged in and has permission to ’read’ the current record. In this example, there is no restriction on downloading images linked by the "small image" field, but we require access control on images linked by the "large image" field.
Access control and Basic authentication Occasionally, it may be necessary to expose actions that have decorators that require access control as services; i.e., to call them from a program or script and still be able to use authentication to check for authorization. Auth enables login via basic authentication:
1 auth.settings.allow_basic_authentication = True With this set, an action like
1 @auth.requires_login() 2 def give_me_time(): 3 import time 4 return time.ctime() can be called, for example, from a shell command:
1 wget --user=[username] --password=[password] 2 http://.../[app]/[controller]/give_me_time Basic login is often the only option for services (described in the next chapter), but it is disabled by default.
Settings and Messages Here is a list of all parameters that can be customized for Auth
1 auth.settings.actions_disabled = [] The actions that should be disabled, for example [’register’].
1 auth.settings.registration_requires_verification = False
Set to True so that registrants receive a verification email and are required to click a link to complete registration. 238 ACCESS CONTROL
1 auth.settings.registration_requires_approval = False
Set to True to prevent login of newly registered users until they are approved (this is done by setting registration key==’’ via appadmin or programmati- cally).
1 auth.settings.create_user_groups = True Set to False if you do not want to automatically create a group for each newly registered user.
1 auth.settings.login_url = URL(r=request, f='user', args='login') Tells web2py the URL of the login page
1 auth.settings.logged_url = URL(r=request, f='user', args='profile') If the user tried to access the register page but is already logged in, he is redirected to this URL.
1 auth.settings.download_url = URL(r=request, f='download') Tells web2py the URL to download uploaded documents. It is necessary to create the profile page in case it contains uploaded files, such as the user image.
1 auth.settings.mailer = None Must point to an object with a send method with the same signature as gluon.tools.Mail.send.
1 auth.settings.captcha = None
Must point to an object with signature similar to gluon.tools.Recaptcha.
1 auth.settings.expiration = 3600 # seconds The expiration time of a login session, in seconds.
1 auth.settings.on_failed_authorization = \ 2 URL (r=request,f='user/on_failed_authorization') The URL to redirect to after a failed authorization.
1 auth.settings.password_field = 'password' Thename of the passwordfield asstored in the db. Theonly reason to change this is when ’password’ is a reserved keyword for the db and so cannot be used as a field name. This is the case, for example, for FireBird.
1 auth.settings.showid = False Determines whether the profile page should show the id of the user.
1 auth.settings.login_next = URL(r=request, f='index') By default, the login page, after successful login, redirects the visitor to the referrer page (if and only if the referrer required login). If there is no referrer, it redirects the visitor to the page pointed to by this variable. AUTHORIZATION 239
1 auth.settings.login_onvalidation = None Function to be called after login validation, but before actual login. The function must take a single argument, the form object.
1 auth.settings.login_onaccept = None Function to be called after login, but before redirection. The function must take a single argument, the form object.
1 auth.settings.login_methods = [auth] Determines alternative login methods, as discussed previously.
1 auth.settings.login_form = auth Sets an alternative login form for single sign-on as discussed previously.
1 auth.settings.allows_basic_auth = False If set to True allows calling actions that have access control enforced through decorators using basic access authentication.
1 auth.settings.logout_next = URL(r=request, f='index') The URL redirected to after logout.
1 auth.settings.register_next = URL(r=request, f='user', args='login') The URL redirected to after registration.
1 auth.settings.register_onvalidation = None Function to be called after registration form validation, but before actual registration, and before any email verification email is sent. The function must take a single argument, the form object.
1 auth.settings.register_onaccept = None Function to be called after registration, but before redirection. The function must take a single argument, the form object.
1 auth.settings.verify_email_next = \ 2 URL (r=request, f='user', args='login') The URL to redirect a visitor to after email address verification.
1 auth.settings.verify_email_onaccept = None Function to be called after completed email verification, but before redirec- tion. The function must take a single argument, the form object.
1 auth.settings.profile_next = URL(r=request, f='index') The URL to redirect visitors to after they edit their profile.
1 auth.settings.retrieve_username_next = URL(r=request, f='index') The URL to redirect visitors to after they request to retrieve their username. 240 ACCESS CONTROL
1 auth.settings.retrieve_password_next = URL(r=request, f='index') The URL to redirect visitors to after they request to retrieve their password.
1 auth.settings.change_password_next = URL(r=request, f='index') The URL to redirect visitors to after they request a new password by email. You can also customize the following messages whose use and context should be obvious:
1 auth.messages.submit_button = 'Submit' 2 auth.messages.verify_password = 'Verify Password' 3 auth.messages.delete_label = 'Check to delete:' 4 auth.messages.function_disabled = 'Function disabled' 5 auth.messages.access_denied = 'Insufficient privileges' 6 auth.messages.registration_verifying = 'Registration needs verification' 7 auth.messages.registration_pending = 'Registration is pending approval' 8 auth.messages.login_disabled = 'Login disabled by administrator' 9 auth.messages.logged_in = 'Logged in' 10 auth.messages.email_sent = 'Email sent' 11 auth.messages.unable_to_send_email = 'Unable to send email' 12 auth.messages.email_verified = 'Email verified' 13 auth.messages.logged_out = 'Logged out' 14 auth.messages.registration_successful = 'Registration successful' 15 auth.messages.invalid_email = 'Invalid email' 16 auth.messages.invalid_login = 'Invalid login' 17 auth.messages.invalid_user = 'Invalid user' 18 auth.messages.is_empty = "Cannot be empty" 19 auth.messages.mismatched_password = "Password fields don't match" 20 auth.messages.verify_email = ... 21 auth.messages.verify_email_subject = 'Password verify' 22 auth.messages.username_sent = 'Your username was emailed to you' 23 auth.messages.new_password_sent = ... 24 auth.messages.password_changed = 'Password changed' 25 auth.messages.retrieve_username = ... 26 auth.messages.retrieve_username_subject = 'Username retrieve' 27 auth.messages.retrieve_password = ... 28 auth.messages.retrieve_password_subject = 'Password retrieve' 29 auth.messages.profile_updated = 'Profile updated' 30 auth.messages.new_password = 'New password' 31 auth.messages.old_password = 'Old password' 32 auth.messages.register_log = 'User %(id)s Registered' 33 auth.messages.login_log = 'User %(id)s Logged-in' 34 auth.messages.logout_log = 'User %(id)s Logged-out' 35 auth.messages.profile_log = 'User %(id)s Profile updated' 36 auth.messages.verify_email_log = ... 37 auth.messages.retrieve_username_log = ... 38 auth.messages.retrieve_password_log = ... 39 auth.messages.change_password_log = .. 40 auth.messages.add_group_log = 'Group %(group_id)s created' 41 auth.messages.del_group_log = 'Group %(group_id)s deleted' 42 auth.messages.add_membership_log = None 43 auth.messages.del_membership_log = None 44 auth.messages.has_membership_log = None 45 auth.messages.add_permission_log = None CENTRAL AUTHENTICATION SERVICE 241
46 auth.messages.del_permission_log = None 47 auth.messages.has_permission_log = None
add|del|has membership logs allow the use of "%(user id)s" and "%(group id)s". add|del|has permission logs allow the use of "%(user id)s", "%(name)s", "%(table name)s", and "%(record id)s".
8.3 Central Authentication Service
web2py provides support for authentication and authorization via appli- ances. Here we discuss the cas appliance for Central Authentication Service (CAS). Notice that at the time of writing CAS is distict and does not work with Auth. This will change in the future. CAS is an open protocol for distributed authentication and it works in the following way: When a visitor arrives at our web site, our application check in the session if the user is already authenticated (for example via a session.token object). If the user is not authenticated, the controller redirects the visitor from the CAS appliance, where the user can log in, register, and manage his credentials (name, email and password). If the user registers, he receives an email, and registration is not complete until he responds to the email. Once the user has successfully registered and logged in, the CAS appliance redirects the user to our application together with a key. Our application uses the key to get the credentials of the user via an HTTP request in the background to the CAS server. Using this mechanism, multiple applications can use the a single sign- on via a single CAS server. The server providing authentication is called a service provider. Applications seeking to authenticate visitors are called service consumers. CAS is similar to OpenID, with one main difference. In the the case of OpenID, the visitor chooses the service provider. In the case of CAS, our application makes this choice, making CAS more secure. You can run only the consumer, only the provider, or both (in a single or separate applications). To run CAS as consumer you must download the file:
1 https://www.web2py.com/cas/static/cas.py and store it as a model file called "cas.py". Then you must edit the controllers that need authentication (for example "default.py") and, at the top, add the following code:
1 CAS.login_url='https://www.web2py.com/cas/cas/login' 2 CAS.check_url='https://www.web2py.com/cas/cas/check' 242 ACCESS CONTROL
3 CAS.logout_url='https://www.web2py.com/cas/cas/logout' 4 CAS.my_url='http://127.0.0.1:8000/myapp/default/login' 5 6 if not session.token and not request.function=='login': 7 redirect(URL(r=request,f='login')) 8 def login(): 9 session.token=CAS.login(request) 10 id,email,name=session.token 11 return dict() 12 def logout(): 13 session.token=None 14 CAS.logout() You mustedit theattributes ofthe CASobjectabove. Bydefault, they point to the CAS provider that runs on "https://mdp.cti.depaul.edu". We provide this service mainly for testing purposes. The CAS.my url has to be the full URL to the login action defined in your application and shown in the code. The CAS provider needs to redirect your browser to this action. Our CAS provider returns a token containing a tuple (id, email, name), where id is the unique record id of the visitor (as assigned by the provider’s database), email is the email address of the visitor (as declared by the visitor to the provider and verified by the provider), and name is the name of the visitor (it is chosenby the visitor and there is no guaranteethis is a realname). If you visit the local url:
1 /myapp/default/login you get redirected to the CAS login page:
1 https://mdp.cti.depaul.edu/cas/cas/login which looks like this:
You may also use third-party CAS services, but you may need to edit line 10 above, since different CAS providers may return tokens containing different values. Check the documentation of the CAS service you need to access for details. Most services only return (id, username). CENTRAL AUTHENTICATION SERVICE 243
After a successful login, you are redirected to the local login action. The view of the local login action is executed only after a successful CAS login. You can download the CAS provider appliance from ref. [32] and run it yourself. If you choose to do so, you must also edit the first lines of the "email.py" model in the appliance, so that it points to your SMPT server. You can also merge the files of the CAS provider appliance provider with those of your application (models under models, etc.) as long there is no filename conflict.
CHAPTER 9
SERVICES
The W3C defines a web service as “a software system designed to support in- teroperable machine-to-machine interaction over a network”. This is a broad definition, and it encompass a large number of protocols not designed for machine-to-human communication, but for machine-to-machine communi- cation such as XML, JSON, RSS, etc. web2py provides, out of the box, support for the many protocols, includ- ing XML, JSON, RSS, CSV,XMLRPC, JSONRPC, AMFRPC. web2py can also be extended to support additional protocols. Each of those protocols is supported in multiple ways, and we make a distinction between: • Rendering the output of a function in a given format (for example XML, JSON, RSS, CSV) • Remote Procedure Calls (for example XMLRPC, JSONRPC, AM- FRPC)
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 245 Copyright © 2009 246 SERVICES
9.1 Rendering a dictionary
HTML, XML, and JSON Consider the following action:
1 def count(): 2 session.counter = (session.counter or 0) + 1 3 return dict(counter=session.counter, now=request.now) This action returns a counterthat is increasedby one whena visitor reloads the page, and the timestamp of the current page request. Normally this page would be requested via:
1 http://127.0.0.1:8000/app/default/count and rendered in HTML. Without writing one line of code, we can ask web2py to render this page using a different protocols by adding an extension to the URL:
1 http://127.0.0.1:8000/app/default/count.html 2 http://127.0.0.1:8000/app/default/count.xml 3 http://127.0.0.1:8000/app/default/count.json The dictionary returned by the action will be rendered in HTML, XML and JSON, respectively. Here is the XML output:
1
1 { 'counter':3, 'now':'2009-08-01 13:00:00' } Notice that date, time, and datetime objects are rendered as strings in ISO format. This is not part of the JSON standard but a web2py convention.
How it works When, for example, the ".xml" extension is called, web2py looks for a tem- plate file called "default/count.xml", and if it does not find it, web2py looks for a template called "generic.xml". The files "generic.html, "generic.xml", "generic.json" are provided with the current scaffolding application. Other extensions can be easily defined by the user. Nothing needs to be done to enable this in a web2py app. To useit in an older web2py app, you may need to copy the "generic.*" files from a later scaffolding app (after version 1.60). RENDERING A DICTIONARY 247
Here is the code for "generic.html"
1 {{extend 'layout.html'}} 2 3 {{=BEAUTIFY(response._vars)}} 4 5 6 7
request
{{=BEAUTIFY(request) }}session
{{=BEAUTIFY(session) }}response
{{=BEAUTIFY( response)}}1 {{ 2 try: 3 from gluon.serializers import xml 4 response.write(xml(response._vars),escape=False) 5 response.headers['Content-Type']='text/xml' 6 except: 7 raise HTTP(405,'no xml') 8 }} And here is the code for "generic.json"
1 {{ 2 try: 3 from gluon.serializers import json 4 response.write(json(response._vars),escape=False) 5 response.headers['Content-Type']='text/json' 6 except: 7 raise HTTP(405,'no json') 8 }} Every dictionary can be rendered in HTML, XML and JSON as long as it only contains python primitive types (int, float, string, list, tuple, dictionary). response. vars contains the dictionary returned by the action. If the dictionary contains other user-defined or web2py-specific objects, they must be rendered by a custom view.
Rendering Rows If you need to render a set of Rows as returned by a select in XML or JSON or another format, first transform the Rows object into a list of dictionaries using the as list() method. Consider for example the following mode: 248 SERVICES
1 db.define_table('person', Field('name')) The following action can be rendered in HTML but not in XML or JSON:
1 def everybody(): 2 people = db().select(db.person.ALL) 3 return dict(people=people) While the following action can rendered in XML and JSON.
1 def everybody(): 2 people = db().select(db.person.ALL).as_list() 3 return dict(people=people)
Custom Formats If, for example, you want to render an action as a Python pickle:
1 http://127.0.0.1:8000/app/default/count.pickle you just need to create a new view file "default/count.pickle" that contains:
1 {{ 2 import cPickle 3 response.headers['Content-Type'] = 'application/python.pickle' 4 response.write(cPickle.dumps(response._vars),escape=False) 5 }} If you want to be able to render as a picked file any action, you only need to save the above file with the name "generic.pickle". Not all objects are pickleable, and not all pickled objects can be unpickled. It is safe to stick to primitive Python files and combinations of them. Objects that do not contain references to file streams or database connections are are usually pickleable, but they can only be unpickled in an environment where the classes of all pickled objects are already defined.
RSS web2py includes a "generic.rss" view that can render the dictionary returned by the action as an RSS feed. Because the RSS feeds have a fixed structure (title, link, description, items, etc.) then for this to work, the dictionary returned by the action must have the proper structure:
1 {'title' : '', 2 'link' : '', 3 'description': '', 4 'created_on' : '', 5 'entries' : []} RENDERING A DICTIONARY 249
end each entry in entries must have the same similar structure:
1 {'title' : '', 2 'link' : '', 3 'description': '', 4 'created_on' : ''} For example the following action can be rendered as an RSS feed:
1 def feed(): 2 return dict(title="my feed", 3 link="http://feed.example.com", 4 description="my first feed", 5 entries=[ 6 dict(title="my feed", 7 link="http://feed.example.com", 8 description="my first feed") 9 ]) by simply visiting the URL:
1 http://127.0.0.1:8000/app/default/feed.rss Alternatively, assuming the following model:
1 db.define_table('rss_entry', 2 Field('title'), 3 Field('link'), 4 Field('created_on','datetime'), 5 Field('description')) the following action can also be rendered as an RSS feed:
1 def feed(): 2 return dict(title=''my feed'', 3 link=''http://feed.example.com'', 4 description=''my first feed'', 5 entries=db().select(db.rss_entry.ALL).as_list())
The as list() method of a Rows object converts the rows into a list of dictionaries. If additional dictionary items are found with key names not explicitly listed here, they are ignored. Here is the "generic.rss" view provided by web2py:
1 {{ 2 try: 3 from gluon.serializers import rss 4 response.write(rss(response._vars),escape=False) 5 response.headers['Content-Type']='application/rss+xml' 6 except: 7 raise HTTP(405,'no rss') 8 }} As one more example of an RSS application, we consider an RSS aggre- gator that collects data from the "slashdot" feed and returns a new web2py feed. 250 SERVICES
1 def aggregator(): 2 import gluon.contrib.feedparser as feedparser 3 d = feedparser.parse( 4 "http://rss.slashdot.org/Slashdot/slashdot/to") 5 return dict(title=d.channel.title, 6 link = d.channel.link, 7 description = d.channel.description, 8 created_on = request.now, 9 entries = [ 10 dict(title = entry.title, 11 link = entry.link, 12 description = entry.description, 13 created_on = request.now) for entry in d.entries]) It can be accessed at:
1 http://127.0.0.1:8000/app/default/aggregator.rss
CSV The Comma Separated Values (CSV) format is a protocol to represent tabular data. Consider the following model:
1 db.define_model('animal', 2 Field('species'), 3 Field('genus'), 4 Field('family')) and the following action:
1 def animals(): 2 animals = db().select(db.animal.ALL) 3 return dict(animals=animals) web2py does not provide a "generic.csv"; you must define a custom view "default/animals.csv" that serializes the animals into CSV. Here is a possible implementation:
1 {{ 2 import cStringIO 3 stream=cStringIO.StringIO() 4 animals.export_to_csv_file(stream) 5 response.headers['Content-Type']='application/vnd.ms-excel' 6 response.write(stream.getvalue(), escape=False) 7 }} Notice that for CSV one could also define a "generic.csv" file, but one would have to specify the name of the object to be serialized ("animals" in the example). This is why we do not provide a "generic.csv" file. REMOTE PROCEDURE CALLS 251
9.2 Remote Procedure Calls
web2py provides a mechanism to turn any function into a web service. The mechanism described here differs from the mechanism described before because:
• The function may take arguments • Thefunctionmaybedefinedinamodeloramoduleinsteadofcontroller • You may want to specify in detail which RPC method should be sup- ported • It enforces a more strict URL naming convention • It is smarter then the previous methods because it works for a fixed set of protocols. For the same reason it is not as easily extensible.
To use this feature: First, you must import and instantiate a service object.
1 from gluon.tools import Service 2 service = Service(globals())
This is already done in the "db.py" model file in the scaffolding application.
Second, you must expose the service handler in the controller:
1 def call(): 2 session.forget() 3 return service()
This already done in the "default.py" controller of the scaffolding application. Remove session.forget() is you plan to usesession cookies with the services.
Third, you must decorate those functions you want to expose as a service. Here is a list of currently supported decorators:
1 @service.run 2 @service.xml 3 @service.json 4 @service.rss 5 @service.csv 6 @service.xmlrpc 7 @service.jsonrpc 8 @service.amfrpc3('domain') As an example consider the following decorated function: 252 SERVICES
1 @service.run 2 def concat(a,b): 3 return a+b This function can be defined in a model or in a controller. This function can now be called remotely in two ways:
1 http://127.0.0.1:8000/app/default/call/run/concat?a=hello&b=world 2 http://127.0.0.1:8000/app/default/call/run/concat/hello/world In both cases the http request returns:
1 helloworld
If the @service.xml decorator is used, the function can be called via
1 http://127.0.0.1:8000/app/default/call/xml/concat?a=hello&b=world 2 http://127.0.0.1:8000/app/default/call/xml/concat/hello/world and the output is returned as XML:
1
1 http://127.0.0.1:8000/app/default/call/json/concat?a=hello&b=world 2 http://127.0.0.1:8000/app/default/call/json/concat/hello/world and the output returned as JSON. If the @service.csv decoratoris used, the service handler requires, as return value, an iterable object of iterable objects, such as a list of lists. Here is an example:
1 @service.csv 2 def table1(a,b): 3 return [[a,b],[1,2]] This service can be called by visiting one of the following URLs:
1 http://127.0.0.1:8000/app/default/call/csv/table1?a=hello&b=world 2 http://127.0.0.1:8000/app/default/call/csv/table1/hello/world and it returns:
1 hello,world 2 1,2
The @service.rss decorator expects a return value in the same format as the "generic.rss" view discussed in the previous section. Multiple decorators are allowed for each function. So far, everything discussed in this section is simply an alternative to the methoddescribedin the previoussection. Thereal powerofthe serviceobject comes with XMLRPC, JSONRPC and AMFRPC, as discussed below. REMOTE PROCEDURE CALLS 253
XMLRPC Consider the following code, for example, in the "default.py" controller:
1 @service.xmlrpc 2 def add(a,b): 3 return a+b 4 5 @service.xmlrpc 6 def div(a,b): 7 return a+b Now in a python shell you can do
1 >>> from xmlrpclib import ServerProxy 2 >>> server = ServerProxy( 3 'http://127.0.0.1:8000/app/default/call/xmlrpc') 4 >>> print server.add(3,4) 5 7 6 >>> print server.add('hello','world') 7 'helloworld' 8 >>> print server.div(12,4) 9 3 10 >>> print server.div(1,0) 11 ZeroDivisionError: integer division or modulo by zero The Python xmlrpclib module provides a client for the XMLRPC protocol. web2py acts as the server. The client connects to the server via ServerProxy and can remotely call decorated functions in the server. The data (a,b) is passed to the function(s), not via GET/POST variables, but properly encoded in the request body using the XMLPRC protocol, and thus it carries with itself type information (int or string or other). The same is true for the return value(s). Moreover, any exception that happens on the server propagates back to the client. There are XMLRPC libraries for many programming languages (including C, C++, Java, C#, Ruby, and Perl), and they can interoperate with each other. This is one the best methods to create applications that talk to each other, independent of the programming language. The XMLRPC client can also be implemented inside a web2py action so that one action can talk to another web2py application (even within the same installation) using XMLRPC. Beware of session deadlocks in this case. If an action calls via XMLRPC a function in the same app, the caller must release the session lock before the call:
1 session.forget() 2 session._unlock(response)
JSONRPC 254 SERVICES
JSONRPC is very similar to XMLRPC, but uses the JSON based protocol to encode the data instead of XML. As an example of application here, we discuss its usage with Pyjamas. Pyjamas is a Python port of the Google Web Toolkit (originally written in Java). Pyjamas allows to write a client application in Python. Pyjamas translates this code into JavaScript. web2py serves the javascript and communicates with it via AJAX requests originating from the client and triggered by user actions. Here we describe how to make Pyjamas work with web2py. It does not require any additional libraries other than web2py and Pyjamas. We are going to build a simple "todo" application with a Pyjamas client (all JavaScript) that talks to the server exclusively via JSONRPC. Here is how to do it: First, create a new application called "todo". Second, in "models/db.py", enter the following code:
1 db=SQLDB('sqlite://storage.sqlite') 2 db.define_table('todo', Field('task')) 3 4 from gluon.tools import Service # import rpc services 5 service = Service(globals()) Third, in "controllers/default.py", enter the following code:
1 def index(): 2 redirect(URL(r=request,f='todoApp')) 3 4 @service.jsonrpc 5 def getTasks(): 6 todos = db(db.todo.id>0).select() 7 return [(todo.task,todo.id) for todo in todos] 8 9 @service.jsonrpc 10 def addTask(taskFromJson): 11 db.todo.insert(task= taskFromJson) 12 return getTasks() 13 14 @service.jsonrpc 15 def deleteTask (idFromJson): 16 del db.todo[idFromJson] 17 return getTasks() 18 19 def call(): 20 session.forget() 21 return service() 22 23 def todoApp(): 24 return dict() The purpose of each function should be obvious. Fourth, in "views/default/todoApp.html", enter the following code:
1 2
REMOTE PROCEDURE CALLS 2553 5
11 simple todo application 12
13 14 type a new task to insert in db, 15 click on existing task to delete it 16 17 20 21 This view just executes the Pyjamas code in "static/output/todoapp". Code that we have not yet created. Fifth, in "static/TodoApp.py" (notice it is TodoApp, not todoApp!), enter the following client code:1 from pyjamas.ui.RootPanel import RootPanel 2 from pyjamas.ui.Label import Label 3 from pyjamas.ui.VerticalPanel import VerticalPanel 4 from pyjamas.ui.TextBox import TextBox 5 import pyjamas.ui.KeyboardListener 6 from pyjamas.ui.ListBox import ListBox 7 from pyjamas.ui.HTML import HTML 8 from pyjamas.JSONService import JSONProxy 9 10 class TodoApp: 11 def onModuleLoad(self): 12 self.remote = DataService() 13 panel = VerticalPanel() 14 15 self.todoTextBox = TextBox() 16 self.todoTextBox.addKeyboardListener(self) 17 18 self.todoList = ListBox() 19 self.todoList.setVisibleItemCount(7) 20 self.todoList.setWidth("200px") 21 self.todoList.addClickListener(self) 22 self.Status = Label("") 23 24 panel.add(Label("Add New Todo:")) 25 panel.add(self.todoTextBox) 26 panel.add(Label("Click to Remove:")) 27 panel.add(self.todoList) 28 panel.add(self.Status) 29 self.remote.getTasks(self) 30 31 RootPanel().add(panel) 256 SERVICES
32 33 def onKeyUp(self, sender, keyCode, modifiers): 34 pass 35 36 def onKeyDown(self, sender, keyCode, modifiers): 37 pass 38 39 def onKeyPress(self, sender, keyCode, modifiers): 40 """ 41 This function handles the onKeyPress event, and will add the 42 item in the text box to the list when the user presses the 43 enter key. In the future, this method will also handle the 44 auto complete feature. 45 """ 46 if keyCode == KeyboardListener.KEY_ENTER and \ 47 sender == self.todoTextBox: 48 id = self.remote.addTask(sender.getText(),self) 49 sender.setText("") 50 if id<0: 51 RootPanel().add(HTML("Server Error or Invalid Response")) 52 53 def onClick(self, sender): 54 id = self.remote.deleteTask( 55 sender.getValue(sender.getSelectedIndex()),self) 56 if id<0: 57 RootPanel().add( 58 HTML("Server Error or Invalid Response")) 59 60 def onRemoteResponse(self, response, request_info): 61 self.todoList.clear() 62 for task in response: 63 self.todoList.addItem(task[0]) 64 self.todoList.setValue(self.todoList.getItemCount()-1, 65 task[1]) 66 67 def onRemoteError(self, code, message, request_info): 68 self.Status.setText("Server Error or Invalid Response: " \ 69 + "ERROR " + code + "-" + message) 70 71 class DataService(JSONProxy): 72 def __init__(self): 73 JSONProxy.__init__(self, "../../default/call/jsonrpc", 74 ["getTasks", "addTask","deleteTask"]) 75 76 if __name__ == '__main__': 77 app = TodoApp() 78 app.onModuleLoad() Sixth, run Pyjamas before serving the application:
1 cd /path/to/todo/static/ 2 python ˜/python/pyjamas-0.5p1/bin/pyjsbuild TodoApp.py This will translate the Pythoncodeinto JavaScriptso thatit can be executed in the browser. To access this application, visit the URL REMOTE PROCEDURE CALLS 257
1 http://127.0.0.1:8000/todo/default/todoApp
Credits This subsection was created by Chris Prinos with help form Luke Kenneth Casson Leighton (creators of Pyjamas) and updated by Alexei Vini- diktov. It has been tested by Pyjamas 0.5p1. The example was inspired by this Django page:
1 http://gdwarner.blogspot.com/2008/10/brief-pyjamas-django-tutorial. html
AMFRPC AMFRPC is the Remote Procedure Call protocol used by Flash clients to communicate with a server. web2py supports AMFRPC but it requires that you run web2py from source and that you preinstall the PyAMF library. This can be installed from the Linux or Windows shell by typing
1 easy_install pyamf (please consult the PyAMF documentation for more details). In this subsection we assume that you are already familiar with Action- Script programming. We will create a simple service that takes two numerical values, adds them together, and returns the sum. We will call our web2py application "pyamf test", and we will call the service addNumbers. First, using Adobe Flash (any version starting from MX 2004), create the Flash client application by starting with a new Flash FLA file. In the first frame of the file, add these lines:
1 import mx.remoting.Service; 2 import mx.rpc.RelayResponder; 3 import mx.rpc.FaultEvent; 4 import mx.rpc.ResultEvent; 5 import mx.remoting.PendingCall; 6 7 var val1 = 23; 8 var val2 = 86; 9 10 service = new Service( 11 "http://127.0.0.1:8000/pyamf_test/default/call/amfrpc3", 12 null, "mydomain", null, null); 13 14 var pc:PendingCall = service.addNumbers(val1, val2); 15 pc.responder = new RelayResponder(this, "onResult", "onFault"); 16 17 function onResult(re:ResultEvent):Void { 18 trace("Result : " + re.result); 19 txt_result.text = re.result; 258 SERVICES
20 } 21 22 function onFault(fault:FaultEvent):Void { 23 trace("Fault: " + fault.fault.faultstring); 24 } 25 26 stop(); This code allows the Flash client to connect to a service that corresponds to a function called "addNumbers" in the file "/pyamf test/default/gateway". You must also import ActionScript version 2 MX remoting classes to enable Remoting in Flash. Add the path to these classes to the classpath settings in the Adobe Flash IDE, or just place the "mx" folder next to the newly created file. Notice the arguments of the Service constructor. The first argument is the URLcorrespondingtotheservicethatwewantwillcreate. Thethirdargument is the domain of the service. We choose to call this domain "mydomain". Second, create a dynamic text field called "txt result" and place it on the stage. Third, you need to set up a web2py gateway that can communicate with the Flash client defined above. Proceed by creating a new web2py app called pyamf test that will host the new service and the AMF gateway for the flash client. Edit the "default.py" controller and make sure it contains
1 @service.amfrpc3('mydomain') 2 def addNumbers(val1, val2): 3 return val1 + val2 4 5 def call(): return service()
Fourth, compile and export/publish the SWF flash client as pyamf test.swf, place the "pyamf test.amf", "pyamf test.html", "AC RunActiveContent.js", and "crossdomain.xml" files in the "static" folder of the newly created appli- ance that is hosting the gateway, "pyamf test". You can now test the client by visiting:
1 http://127.0.0.1:8000/pyamf_test/static/pyamf_test.html The gateway is called in the background when the client connects to addNumbers. If you are suing AMF0 instead of AMF3 you can also use the decorator:
1 @service.amfrpc instead of:
1 @service.amfrpc3('mydomain') In this case you also need to change the service URL to: LOW LEVEL API AND OTHER RECIPES 259
1 http://127.0.0.1:8000/pyamf_test/default/call/amfrpc
9.3 Low Level API and Other Recipes
simplejson web2py includes gluon.contrib.simplejson, developedby Bob Ippolito. This module provides the most standard Python-JSON encoder-decoder. SimpleJSON consists of two functions:
• gluon.contrib.simplesjson.dumps(a) encodes a Python object a into JSON.
• gluon.contrib.simplejson.loads(b) decodes a JavaScript object b into a Python object.
Object types that can be serialized include primitive types, lists, and dictio- naries. Compoundobjects canbe serializedwith the exceptionof user defined classes. Here is a sample action (for example in controller "default.py") that seri- alizes the Python list containing weekdays using this low level API:
1 def weekdays(): 2 names=['Sunday','Monday','Tuesday','Wednesday', 3 'Thursday','Friday','Saturday'] 4 import gluon.contrib.simplejson 5 return gluon.contrib.simplejson.dumps(names)
Below is a sample HTML page that sends an Ajax request to the above action, receives the JSON message, and stores the list in a corresponding JavaScript variable:
1 {{extend 'layout.html'}} 2
The code uses the jQuery function $.getJSON, which performs the Ajax call and, on response, stores the weekdays names in a local JavaScript variable data and passes the variable to the callback function. In the example the callback function simply alerts the visitor that the data has been received. 260 SERVICES
PyRTF Another common need of web sites is that of generating Word-readable text documents. The simplest way to do so is using the Rich Text Format (RTF) document format. This format was invented by Microsoft and it has since become a standard. web2py includes gluon.contrib.pyrtf, developed by Simon Cusack and re- vised by Grant Edwards. This module allows you to generate RTF documents programmatically including colored formatted text and pictures. In the following example we instantiate two basic RTF classes, Document and Section, append the latter to the former and insert some dummy text in the latter:
1 def makertf(): 2 import gluon.contrib.pyrtf as q 3 doc=q.Document() 4 section=q.Section() 5 doc.Sections.append(section) 6 section.append('Section Title') 7 section.append('web2py is great. '*100) 8 response.headers['Content-Type']='text/rtf' 9 return q.dumps(doc)
In the end the Document is serialized by q.dumps(doc). Notice that before returning an RTF document it is necessary to specify the content-type in the header else the browser does not know how to handle the file. Depending on the configuration, the browser may ask you whether to save this file or open it using a text editor.
ReportLab and PDF web2py can also generate PDF documents, with an additional library called "ReportLab"[66]. If you are running web2py from source, it is sufficient to have ReportLab installed. If you are running the Windows binary distribution, you need to unzip ReportLab in the "web2py/" folder. If you are running the Mac binary distribution, you need to unzip ReportLab in the folder:
1 web2py.app/Contents/Resources/ From now on we assume ReportLab is installed and that web2py can find it. We will create a simple action called "get me a pdf" that generates a PDF document.
1 from reportlab.platypus import * 2 from reportlab.lib.styles import getSampleStyleSheet 3 from reportlab.rl_config import defaultPageSize 4 from reportlab.lib.units import inch, mm SERVICES AND AUTHENTICATION 261
5 from reportlab.lib.enums import TA_LEFT, TA_RIGHT, TA_CENTER, TA_JUSTIFY 6 from reportlab.lib import colors 7 from uuid import uuid4 8 from cgi import escape 9 import os 10 11 def get_me_a_pdf(): 12 title = "This The Doc Title" 13 heading = "First Paragraph" 14 text = 'bla '* 10000 15 16 styles = getSampleStyleSheet() 17 tmpfilename=os.path.join(request.folder,'private',str(uuid4())) 18 doc = SimpleDocTemplate(tmpfilename) 19 story = [] 20 story.append(Paragraph(escape(title),styles["Title"])) 21 story.append(Paragraph(escape(heading),styles["Heading2"])) 22 story.append(Paragraph(escape(text),styles["Normal"])) 23 story.append(Spacer(1,2*inch)) 24 doc.build(story) 25 data = open(tmpfilename,"rb").read() 26 os.unlink(tmpfilename) 27 response.headers['Content-Type']='application/pdf' 28 return data
Notice how we generate the PDF into a unique temporaty file, tmpfilename, we read the generated PDF from the file, then we deletedthe file. For more information about the ReportLab API, refer to the ReportLab documentation. We strongly recomment using the Platypus API of Report- Lab, such as Paragraph, Spacer, etc.
9.4 Services and Authentication
In the previous chapter we have discussedthe use of the following decorators:
1 @auth.requires_login() 2 @auth.requires_memebrship(...) 3 @auth.requires_permission(...) For normal actions (not decorated as services), these decorators can be used even if the output is rendered in a format other than HTML. For functions defined as services and decorated using the @service... decorators, the @auth... decorators should not be used. The two types of decorators cannot be mixed. If authenticaiton is to be performed, it is the call actions that needs to be decorated:
1 @auth.requires_login() 2 def call(): return service() 262 SERVICES
Notice that it also possible to instantiate multiple service objects, regis- ter the same different functions with them, and expose some of them with authentication and some not:
1 public_services=Service(globals()) 2 private_services=Service(globals()) 3 4 @public_service.jsonrpc 5 @private_service.jsonrpc 6 def f(): return 'public' 7 8 @private_service.jsonrpc 9 def g(): return 'private' 10 11 def public_call(): return public_service() 12 13 @auth.requires_login() 14 def private_call(): return private_service() This assumes that the caller is passing credentials in the HTTP header (a valid sessioncookie or using basic authentication, as discussedin the previous section). The client must support it; not all clients do. CHAPTER 10
AJAX RECIPES
While web2py is mainly for server-side development, it comes with the base jQuery library [31], jQuery calendars (date picker, datetime picker and clock) and some additional JavaScript functions based on jQuery. Nothing in web2py prevents you from using other Ajax [67] libraries such as Prototype, Scriptaculous or ExtJS but we decided to package jQuery because we find it to be easier to use and more powerful than any other equivalent libraries. We also find it captures the web2py spirit of being functional and concise.
10.1 web2py ajax.html
The scaffolding web2py application "welcome" includes a file called
1 views/web2py_ajax.html This file is included in the HEAD of the default "layout.html" and it provides the following services:
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 263 Copyright © 2009 264 AJAX RECIPES
• Includes static/jquery.js.
• Includes static/calendar.js and static/calendar.css, if they exist.
• Defines a popup function.
• Defines a collapse function (based on jQuery slideToggle).
• Defines a fade function (based on jQuery fade).
• Defines an ajax function (based on jQuery $.ajax).
• Makes any DIV of class "error" or any tag object of class "flash" slide down.
• Prevents typing invalid integers in INPUT fields of class "integer".
• Prevents typing invalid floats in INPUT fields of class "double".
• Connects INPUT fields of type "date" with a popup date picker.
• ConnectsINPUTfieldsoftype"datetime"withapopupdatetimepicker.
• Connects INPUT fields of type "time" with a popup time picker.
popup, collapse, and fade are included only for backward compatibility, and are not discussed here. Here is an an example of how the other effects play well together. Consider a test app with the following model:
1 db = DAL("sqlite://db.db") 2 db.define_table('mytable', 3 Field('field_integer', 'integer'), 4 Field('field_date', 'date'), 5 Field('field_datetime', 'datetime'), 6 Field('field_time', 'time')) with this "default.py" controller:
1 def index(): 2 form = SQLFORM(db.mytable) 3 if form.accepts(request.vars, session): 4 response.flash = 'record inserted' 5 return dict(form=form) and the following "default/index.html" view:
1 {{extend 'layout.html}} 2 {{=form}} WEB2PY AJAX.HTML 265
The "index" action generates the following form:
If an invalid form is submitted, the server returns the page with a modified form containing error messages. The error messages are DIVs of class "error", and because of the above web2py ajax code, the errors appears with a slide-down effect:
The color of the errors is given in the CSS code in "layout.html". 266 AJAX RECIPES
The web2py ajax code prevents you from typing an invalid value in the input field. This is done before and in addition to, not as a substitute for, the server-side validation. The web2py ajax code displays a date picker when you enter an INPUT field of class "date", and it displays a datetime picker when you enter an INPUT field of class "datetime". Here is an example:
The web2py ajax code also displays the following time picker when you try to edit an INPUT field of class "time": WEB2PY AJAX.HTML 267
Upon submission, the controller action sets the response flash to the mes- sage "record inserted". The default layout renders this message in a DIV with id="flash". The web2py ajax code is responsible for making this DIV slide down and making it disappear when you click on it:
These and other effects are accessible programmatically in the views and via helpers in controllers. 268 AJAX RECIPES
10.2 jQuery Effects
Using jQuery effects is very easy. Here we describe how to do it. The basic effects described here do not require any additional files; every- thing you need is already included for you by web2py ajax.html. HTML/XHTML objects can be identified by their type (for example a DIV), their classes, or their id. For example:
1
1 jQuery('.one') // address object by class "one" 2 jQuery('#a') // address object by id "a" 3 jQuery('DIV.one') // address by object of type "DIV" with class "one" 4 jQuery('DIV #a') // address by object of type "DIV" with id "a" and to the latter with
1 jQuery('.two') 2 jQuery('#b') 3 jQuery('DIV.two') 4 jQuery('DIV #b') or you can refer to both with
1 jQuery('DIV') Tag objects are associated to events, such as "onclick". jQuery allows linking these events to effects, for example "slideToggle":
1
1
A common situation is the need to execute some JavaScript code only after the entire document has been loaded. This is usually done by the onload attribute of BODY but jQuery provides an alternative way that does not require editing the layout:
1
Form Events • onchange: Script to be run when the element changes • onsubmit: Script to be run when the form is submitted • onreset: Script to be run when the form is reset • onselect: Script to be run when the element is selected • onblur: Script to be run when the element loses focus • onfocus: Script to be run when the element gets focus
Keyboard Events • onkeydown: Script to be run when key is pressed • onkeypress: Script to be run when key is pressed and released • onkeyup: Script to be run when key is released
Mouse Events • onclick: Script to be run on a mouse click • ondblclick: Script to be run on a mouse double-click • onmousedown: Script to be run when mouse button is pressed • onmousemove: Script to be run when mouse pointer moves • onmouseout: Script to be run when mouse pointer moves out of an element 270 AJAX RECIPES
• onmouseover: Script to be run when mouse pointer moves over an element
• onmouseup: Script to be run when mouse button is released
Here is a list of useful effects defined by jQuery:
Effects
• jQuery(...).attr(name): Returns the name of the attribute value
• jQuery(...).attr(name, value): Sets the attribute name to value
• jQuery(...).show(): Makes the object visible
• jQuery(...).hide(): Makes the object hidden
• jQuery(...).slideToggle(speed, callback): Makes the object slide up or down
• jQuery(...).slideUp(speed, callback): Makes the object slide up
• jQuery(...).slideDown(speed, callback): Makes the object slide down
• jQuery(...).fadeIn(speed, callback): Makes the object fade in
• jQuery(...).fadeOut(speed, callback): Makes the object fade out
The speed argument is usually "slow", "fast" or omitted (the default). The callback is an optional function that is called when the effect is completed. jQuery effects can also easily be embedded in helpers, for example, in a view:
1 {{= DIV('clickme!', _onclick="jQuery(this).fadeOut()")}}
jQuery is a very compact and concise Ajax library; therefore web2py does not need an additional abstraction layer on top of jQuery (except for the ajax function discussed below). The jQuery APIs are accessible and readily available in their native form when needed. Consult the documentation for more information about these effects and other jQuery APIs. The jQuery library can also be extended using plugins and User Interface Widgets. This topic is not covered here; see ref. [69] for details. JQUERY EFFECTS 271
Conditional Fields in Forms A typical application of jQuery effects is a form that changes its appearance based on the value of its fields. This is easy in web2py because the SQLFORM helper generates forms that are "CSS friendly". The form contains a table with rows. Each row contains a label, an input field, and an optional third column. The items have ids derived strictly from the name of the table and names of the fields. The convention is that every INPUT field has a name equal to table- name fieldname and is contained in a row called tablename fieldname row. As an example, create an input form that asks for a taxpayer’s name and for the name of the taxpayer’s spouse, but only if he/she is married. Create a test application with the following model:
1 db = DAL('sqlite://db.db') 2 db.define_table('taxpayer', 3 Field('name'), 4 Field('married', 'boolean'), 5 Field('spouse_name'))
the following "default.py" controller:
1 def index(): 2 form = SQLFORM(db.taxpayer) 3 if form.accepts(request.vars, session): 4 response.flash = 'record inserted' 5 return dict(form=form)
and the following "default/index.html" view:
1 {{extend 'layout.html'}} 2 {{=form}} 3
The script in the view has the effect of hiding the row containing the spouse’s name: 272 AJAX RECIPES
When the taxpayer checks the "married" checkbox, the spouse’s name field reappears:
Here "taxpayer married" is the checkbox associated to the "boolean" field "married" of table "taxpayer". "taxpayer spouse name row" it the row con- taining the input field for "spouse name" of table "taxpayer".
Confirmation on Delete Anotheruseful application is requiring confirmation when checkinga "delete" checkbox such as the delete checkbox that appears in edit forms. Consider the above example and add the following controller action:
1 def edit(): 2 row = db(db.taxpayer.id==request.args[0]).select()[0] 3 form = SQLFORM(db.taxpayer, row, deletable=True) JQUERY EFFECTS 273
4 if form.accepts(request.vars, session): 5 response.flash = 'record updated' 6 return dict(form=form)
and the corresponding view "default/edit.html"
1 {{extend 'layout.html'}} 2 {{=form}}
The deletable=True argumentin the SQLFORMconstructorinstructs web2py to display a "delete" checkbox in the edit form. web2py’s "web2py ajax.html" includes the following code:
1 jQuery(document).ready(function(){ 2 jQuery('input.delete').attr('onclick', 3 'if(this.checked) if(!confirm( 4 "{{=T('Sure you want to delete this object?')}}")) 5 this.checked=false;'); 6 });
By convention this checkbox has a class equal to "delete". The jQuery code above connects the onclick event of this checkbox with a confirmation dialog (standard in JavaScript) and unchecks the checkbox if the taxpayer does not confirm: 274 AJAX RECIPES
10.3 The ajax Function
In web2py ajax.html, web2py defines a function called ajax which is based on, but should not be confused with, the jQuery function $.ajax. The latter is much more powerful than the former, and for its usage, we refer you to ref. [31] and ref. [68]. However, the former function is sufficient for many complex tasks, and is easier to use. The ajax function is a JavaScript function that has the following syntax:
1 ajax(url, [id1, id2, ...], target) It asynchronously calls the url (first argument), passes the values of the fields with the id equal to one of the ids in the list (second argument), then stores the response in the innerHTML of the tag with the id equal to target (the third argument). Here is an example of a default controller:
1 def one(): 2 return dict() 3 4 def echo(): 5 return request.vars.name and the associated "default/one.html" view:
1 {{extend 'layout.html'}} 2
5 When you type something in the INPUT field, as soon as you release a key (onkeyup), the ajax function is called, and the value of the id="name" field is passed to the action "echo", which sends the text back to the view. The ajax function receives the response and displays the echo response in the "target" DIV.Eval target The thrid argument of the ajax function can be the string ":eval". This means that the string returned by server will not be embedded in the document but it will be evaluated instead. Here is an example of a default controller:
1 def one(): 2 return dict() 3 4 def echo(): 5 return "jQuery('#target').html(%s);" % repr(request.vars.name) THE AJAX FUNCTION 275
and the associated "default/one.html" view:
1 {{extend 'layout.html'}} 2
5 This allows for more articulated responses than simple strings.Auto-completion
Another application of the above ajax function is auto-completion. Here we wish to create an input field that expects a month name and, when the visitor types an incomplete name, performs auto-completion via an Ajax request. In response, an auto-completion drop-box appears below the input field. This can be achieved via the following default controller:
1 def month_input(): 2 return dict() 3 4 def month_selector(): 5 if not request.vars.month: 6 return '' 7 months = ['January', 'February', 'March', 'April', 'May', 8 'June', 'July', 'August', 'September' ,'October', 9 'November', 'December'] 10 selected = [m for m in months \ 11 if m.startswith(request.vars.month.capitalize())] 12 return ''.join([DIV(k, 13 _onclick="jQuery('#month').val('%s')" % k, 14 _onmouseover="this.style.backgroundColor='yellow'", 15 _onmouseout="this.style.backgroundColor='white'" 16 ).xml() for k in selected]) and the corresponding "default/month input.html" view:
1 {{extend 'layout.html'}} 2 7 8
13 276 AJAX RECIPESThe jQuery script in the view triggers the Ajax request each time the visitor types something in the "month" input field. The value of the input field is submitted with the Ajax request to the "month selector" action. This action finds a list of month names that start with the submitted text (selected), builds a list of DIVs (each one containing a suggested month name), and returns a string with the serialized DIVs. The view displays the response HTML in the "suggestions" DIV. The "month selector" action generates both the suggestions and the JavaScript code embedded in the DIVs that must be executed when the visitor clicks on each suggestion. For example when the visitor types "Ma" the callback action returns:
1
If the months are stored in a database table such as:
1 db.define_table('month', Field('name'))
then simply replace the month selector action with:
1 def month_input(): 2 return dict() 3 4 def month_selector(): 5 it not request.vars.month: 6 return '' 7 pattern = request.vars.month.capitalize() + '%' 8 selected = [row.name for row in db(db.month.name.like(pattern)). select()] 9 return ''.join([DIV(k, 10 _onclick="jQuery('#month').val('%s')" % k, 11 _onmouseover="this.style.backgroundColor='yellow'", 12 _onmouseout="this.style.backgroundColor='white'" 13 ).xml() for k in selected]) jQuery provides an optional Auto-complete Plugin with additional func- tionalities, but that is not discussed here. THE AJAX FUNCTION 277
Form Submission Here we consider a page that allows the visitor to submit messages using Ajax without reloading the entire page. It contains a form "myform" and a "target" DIV. When the form is submitted, the server may accept it (and perform a database insert) or reject it (because it did not pass validation). The corresponding notification is returned with the Ajax response and displayed in the "target" DIV. Build a test application with the following model:
1 db = DAL('sqlite://db.db') 2 db.define_table('post', Field('your_message', 'text')) 3 db.post.your_message.requires = IS_NOT_EMPTY() Notice that each post has a single field "your message" that is required to be not-empty. Edit the default.py controller and write two actions:
1 def index(): 2 return dict() 3 4 def new_post(): 5 form = SQLFORM(db.post) 6 if form.accepts(request.vars, formname=None): 7 return DIV("Message posted") 8 elif form.errors: 9 return TABLE(*[TR(k, v) for k, v in form.errors.items()]) The first action does nothing other than return a view. The second action is the Ajax callback. It expects the form variables in request.vars, processes them and returns DIV("Message posted") upon success or a TABLE of error messages upon failure. Now edit the "default/index.html" view:
1 {{extend 'layout.html'}} 2 3
4 5 9 10 Notice how in this example the form is created manually using HTML, but it is processed by the SQLFORM in a different action than the one that 278 AJAX RECIPESdisplays the form. The SQLFORM object is never serialized in HTML. SQLFORM.accepts in this case does not take a session and sets formname=None, because we chose not to set the form name and a form key in the manual HTML form. The script at the bottom of the view connects the "myform" submit button to an inline function which submits the INPUT with id="your message" using the web2py ajax function, and displays the answer inside the DIV with id="target".
Voting and Rating AnotherAjax applicationis voting orrating items in a page. Here we consider an application that allows visitors to vote on posted images. The application consistsofasinglepagethatdisplaystheimagessortedaccordingtotheirvote. We will allow visitors to vote multiple times, although it is easy to change this behavior if visitors are authenticated, by keeping track of the individual votes in the database and associating them with the request.env.remote addr of the voter. Here is a sample model:
1 db = DAL('sqlite://images.db') 2 db.define_table('item', 3 Field('image', 'upload'), 4 Field('votes', 'integer', default=0))
Here is the default controller:
1 def list_items(): 2 items = db().select(db.item.ALL, orderby=˜db.item.votes) 3 return dict(items=items) 4 5 def download(): 6 return response.download(request, db) 7 8 def vote(): 9 item = db(db.item.id==request.vars.id).select()[0] 10 new_votes = item.votes + 1 11 item.update_record(votes=new_votes) 12 return str(new_votes) The download action is necessary to allow the list items view to download images stored in the "uploads" folder. The votes action is used for the Ajax callback. Here is the "default/list items.html" view:
1 {{extend 'layout.html'}} 2 3
4 {{for item in items:}} THE AJAX FUNCTION 2795
6 8
9 Votes={{=item.votes}} 10 [vote up] 12
Ajax callbacks can be used to perform computations in the back- ground, but we reccomment using CRON instead (discussed in chapter 4), since the web server enforces a timeout on threads. If the computation takes too long, the web server kills it. Refer to your web server parameters to set the timeout value.
CHAPTER 11
DEPLOYMENT RECIPES
There are multiple ways to deploy web2py in a production environment; the details depend on the configuration and the services provided by the host. In this chapter we consider the following issues: • Configuration of production-quality web servers (Apache, Lighttpd, Cherokee) • Security Issues • Scalability issues • Deployment on the Google App Engine (GAE [12]) web2py comes with an SSL [20] enabled web server, the CherryPy ws- giserver [21]. While this is a fast web server, it has limited configuration capabilities. For this reason it is best to deploy web2py behind Apache [71], Lighttpd [75] or Cherokee [76]. These are free and open-source web servers that are customizable and have been proven to be reliable in high traffic pro- duction environments. They can be configured to serve static files directly, deal with HTTPS, and pass control to web2py for dynamic content.
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 281 Copyright © 2009 282 DEPLOYMENT RECIPES
Until a few years ago, the standard interface for communication be- tween web servers and web applications was the Common Gateway Interface (CGI) [70]. The main problem with CGI is that it creates a new process for each HTTP request. If the web application is written in an interpreted language, each HTTP request served by the CGI scripts starts a new instance of the interpreter. This is slow, and it should be avoided in a production environment. Moreover, CGI can only handle simple responses. It cannot handle, for example, file streaming. web2py provides a file modpythonhandler.py to interface to CGI. One solution to this problem is to use the mod python module for Apache. mod python starts one instance of the Python interpreter when Apache starts, and serves each HTTP request in its own thread without having to restart Python each time. This is a better solution than CGI, but it is not an op- timal solution, since mod python uses its own interface for communication between the web server and the web application. In mod python, all hosted applications run under the same user-id/group-id, which presents security issues. web2py provides a file cgihandler.py to interface to mod python. In the last few years, the Python community has come together behind a new standard interface for communication between web servers and web applications written in Python. It is called Web Server Gateway Interface (WSGI) [17, 18]. web2py was built on WSGI, and it provides handlers for using other interfaces when WSGI is not available. Apache supports WSGI via the module mod wsgi [74] developed by Gra- ham Dumpleton. web2py provides a file wsgihandler.py to interface to WSGI. Some web hosting services do not supportmod wsgi. Inthis case,wemust use Apache as a proxy and forward all incoming requests to the web2py built-in web server (running for example on localhost:8000). In bothcases,with mod wsgi and/ormod proxy, Apachecan be configured to serve static files and deal with SSL encryption directly, taking the burden off web2py. The Lighttpd web server does not currently support the WSGI interface, but it does support the FastCGI [77] interface, which is an improvement over CGI. FastCGI’s main aim is to reduce the overhead associated with interfacing the web server and CGI programs, allowing a server to handle more HTTP requests at once. According to the Lighttpd web site, "Lighttpd powers several popular Web 2.0 sites such as YouTube and Wikipedia. Its high speed IO-infrastructure allows them to scale several times better with the same hardware than with 283 alternative web-servers". Lighttpd with FastCGI is, in fact, faster than Apache with mod wsgi. web2py provides a file fcgihandler.py to interface to FastCGI. web2py also includes a gaehandler.py to interface with the Google App Engine (GAE). On GAE, web applications run "in the cloud". This means that the framework completely abstracts any hardware details. The web application is automatically replicated as many times as necessary to serve all concurrent requests. Replication in this case means more than multiple threads on a single server; it also means multiple processes on different servers. GAE achieves this level of scalability by blocking write access to the file system and all persistent information must be stored in the Google BigTable datastore or in memcache. On non-GAE platforms, scalability is an issue that needs to be addressed, and it may require some tweaks in the web2py applications. The most common way to achieve scalability is by using multiple web servers behind a load-balancer (a simple round robin, or something more sophisticated, receiving heartbeat feedback from the servers). Even if there are multiple web servers, there must be one, and only one, databaseserver. By default, web2py usesthe file systemfor storing sessions, error tickets, uploaded files, and the cache. This means that in the default configuration, the corresponding folders have to be shared folders:
In the rest of the chapter, we consider various recipes that may provide an improvement over this naive approach, including: • Store sessions in the database, in cache or do not store sessions at all. 284 DEPLOYMENT RECIPES
• Store tickets on local filesystems and move them into the database in batches. • Use memcache instead of cache.ram and cache.disk. • Store uploaded files in the database instead of the shared filesystem. While we recommend following the first three recipes, the fourth recipe may provide an advantage mainly in the case of small files, but may be counterproductive for large files.
11.1 Setup Apache on Linux
In this section, we use Ubuntu 8.04 Server Edition as the reference platform. The configuration commands are very similar on other Debian-based Linux distribution, but they may differ for Red Hat-based systems. First, make sure allthe necessary Python andApachepackagesare installed by typing the following shell commands:
1 sudo apt-get update 2 sudo apt-get -y upgrade 3 sudo apt-get -y install openssh-server 4 sudo apt-get -y install python 5 sudo apt-get -y install python-dev 6 sudo apt-get -y install apache2 7 sudo apt-get -y install libapache2-mod-wsgi Then, enable the SSL module, the proxy module, and the WSGI module in Apache:
1 sudo a2enmod ssl 2 sudo a2enmod proxy 3 sudo a2enmod proxy_http 4 sudo a2enmod wsgi Create the SSL folder, and put the SSL certificates inside it:
1 sudo mkdir /etc/apache2/ssl You should obtain your SSL certificates from a trusted Certificate Authority such as verisign.com, but, for testing purposes, you can generate your own self-signed certificates following the instructions in ref. [73] Then restart the web server:
1 sudo /etc/init.d/apache2 restart
The Apache configuration file is:
1 /etc/apache2/sites-available/default SETUP MOD WSGI ON LINUX 285
The Apache logs are in:
1 /var/log/apache2/
11.2 Setup mod wsgi on Linux
Download and unzip web2py source on the machine where you installed the web server above. Install web2py under /users/www-data/, for example, and give ownership to user www-data and group www-data. These steps can be performed with the following shell commands:
1 cd /users/www-data/ 2 sudo wget http://web2py.com/examples/static/web2py_src.zip 3 sudo unzip web2py_src.zip 4 sudo chown -R www-data:www-data /user/www-data/web2py To set up web2py with mod wsgi, create a new Apache configuration file:
1 /etc/apache2/sites-available/web2py and include the following code:
1
30 31 32 CustomLog /private/var/log/apache2/access.log common 33 ErrorLog /private/var/log/apache2/error.log 34 When you restart Apache, it should pass all the requests to web2y without going through the CherryPy wsgiserver. Here are some explanations:
1 WSGIDaemonProcess web2py user=www-data group=www-data 2 display-name=%{GROUP} defines a daemon process group in context of "web2py.example.com". By defining this inside of the virtual host, only this virtual host, including any virtual host for same server name but on a different port, can accessthis using WSGIProcessGroup. The "user" and "group" options shouldbe set to the user who has write access to the directory where web2py was setup. You do not need to set "user" and "group" if you made the web2py installation directory writable to the user that Apache runs as by default. The "display-name" option is so that process name appears in "ps" output as "(wsgi:web2py)" instead of as name of Apache web server executable. As no "processes" or "threads" options specified, the daemon process group will have a single process with 15 threads running within that process. This is usually more than adequate for most sites and should be left as is. If overriding it, do not use "processes=1" as doing so will disable any in browser WSGI debugging tools that check the "wsgi.multiprocess" flag. This is because any use of the "processes" option will cause that flag to be set to true, even if a single process and such tools expect that it be set to false. Note that if your own application code or some third party extension module you are using with Python is not thread safe, instead use options "processes=5 threads=1". This will create five processes in the daemon process group where each process is single threaded. You might consider using "maximum-requests=1000" if your application leaks Python objects through inability for them to be garbage collected properly.
1 WSGIProcessGroup web2py delegates running of all WSGI applications to the daemon process group that was configured using the WSGIDaemonProcess directive.
1 WSGIScriptAlias / /users/www-data/web2py/wsgihandler.py mounts the web2py application. In this case it is mounted at the root of the web site. Not known how to get web2py to mount at a sub URL as doesn"t appeartobeagoodWSGIcitizenandworkoutwhereitismountedfromvalue of SCRIPT NAME and then automatically adjust everything appropriately without further manual user configuration. SETUP MOD WSGI ON LINUX 287
1
1
1
1
1 scripts/web2py-wsgi.conf This section was created with help from Graham Dumpleton, developer of mod wsgi.
mod wsgi and SSL To force some applications (for example admin and appadmin) to go over HTTPS, store the SSL certificate and key files:
1 /etc/apache2/ssl/server.crt 2 /etc/apache2/ssl/server.key
and edit the Apache configuration file web2py.conf and append: 288 DEPLOYMENT RECIPES
1
1 https://www.example.com/admin 2 https://www.example.com/examples/appadmin 3 http://www.example.com/examples but not:
1 http://www.example.com/admin 2 http://www.example.com/examples/appadmin
11.3 Setup mod proxy on Linux
Some Unix/Linux distributions can run Apache,but do not supportmod wsgi. In this case,the simplestsolutionis to run Apacheas a proxyand haveApache deal with static files only. Here is a minimalist Apache configuration:
1 NameVirtualHost *:80 2 ### deal with requests on port 80 SETUP MOD PROXY ON LINUX 289
3
1 nohup python web2py.py -a '
You can specify a password with the -a option or use the "
1 NameVirtualHost *:80 2 NameVirtualHost *:443 3 ### deal with requests on port 80 4
19 ### proxy all the other requests 20
The administrative interface must be disabled when web2py runs on a shared host with mod proxy, or it will be exposed to other users.
11.4 Start as Linux Daemon
Unless you are using mod wsgi, you should setup the web2py server so that it can be started/stopped/restarted as any other Linux daemon, and so it can start automatically at the computer boot stage. The process to set this up is specific to various Linux/Unix distributions. In the web2py folder, there are two scripts which can be used for this purpose:
1 scripts/web2py.ubuntu.sh 2 scripts/web2py.fedora.sh On Ubuntu and other Debian-based Linux distributions, edit the script "web2py.ubuntu.sh" and replace the "/usr/lib/web2py" path with the path of your web2py installation, then type the following shell commands to move the file into the proper folder, register it as a startup service, and start it:
1 sudo cp scripts/web2py.ubuntu.sh /etc/init.d/web2py 2 sudo update-rc.d web2py defaults 3 sudo /etc/init.d/web2py start SETUP APACHE AND MOD WSGI ON WINDOWS 291
On Fedora and other distributions based on Red Hat, edit the script "web2py.fedora.sh" and replace the "/usr/lib/web2py" path with the path of your web2py installation, then type the following shell commands to move the file into the proper folder, register it as a startup service and start it:
1 sudo cp scripts/web2py.fedora.sh /etc/rc.d/init.d/web2pyd 2 sudo chkconfig --add web2pyd 3 sudo service web2py start
11.5 Setup Apache and mod wsgi on Windows
Installing Apache, and mod wsgi under Windows requires a different proce- dure. Here are assuming Python 2.5 is installed, you are running from source and web2py is located at c:/web2py. First download the requires packages:
• Apache apache 2.2.11-win32-x86-openssl-0.9.8i.msi from
1 http://httpd.apache.org/download.cgi
• mod wsgi from
1 http://adal.chiriliuc.com/mod_wsgi/revision_1018_2.3/ mod_wsgi_py25_apache22/mod_wsgi.so
Second, run apache...msi and follow the wizard screens. On the server information screen 292 DEPLOYMENT RECIPES
enter all requested values:
• Network Domain: enter the DNS domain in which your server is or will be registered in. For example, if your server’s full DNS name is server.mydomain.net, you would type mydomain.net here • ServerName: Your server’s full DNS name. From the example above, you would type server.mydomain.net here. Enter a fully qualified do- main name or IP address from the web2py install, not a shortcut, for more information see http://httpd.apache.org/docs/2.2/mod/core.html. • Administrator’s Email Address. Enter the server administrator’s or webmaster’s email address here. This address will be displayed along with error messages to the client by default.
Continue with a typical install to the end unless otherwise required The wizard, by default, installed Apache in the folder:
1 C:/Program Files/Apache Software Foundation/Apache2.2/
From now on we refer to this folder simply as Apache2.2. Third, copy the downloaded mod wsgi.so to Apache2.2/modules The following information about SSL certificates was found in
1 http://port25.technet.com/videos/images/ TechnicalAnalysisInstallingApacheonWindo_C21A/ InstallingApacheonWindows.pdf written by Chris Travers, published by the Open Source Software Lab at Microsoft, December 2007. Fourth, create and place the server.crt and server.key certificates (as created in the previous section) into Apache2.2/conf. Notice the cnf file is in Apache2.2/conf/openssl.cnf. Fifth, edit Apache2.2/conf/httpd.conf, remove the comment mark (the # character) from the line
1 LoadModule ssl_module modules/mod_ssl.so add the following line after all the other LoadModule lines
1 LoadModule wsgi_module modules/mod_wsgi.so look for "Listen 80" and add this line after it
1 Listen 443 append the following lines at the end changing drive letter, port number, ServerName according to your values
1 NameVirtualHost *:443 2
4 ServerName server1 5 6
11.6 Start as Windows Service
What Linux calls a daemon, Windows calls a service. The web2py server can easily be installed/started/stopped as a Windows service. In order to use web2py as a Windows service, you must create a file "options.py" with startup parameters:
1 import socket, os 2 ip = socket.gethostname() 3 port = 80 294 DEPLOYMENT RECIPES
4 password = '
1 python web2py.py -W install and start/stop the service with:
1 python web2py.py -W start 2 python web2py.py -W stop
11.7 Setup Lighttpd
Youcan installLighttpd on a Ubuntu orother Debian-basedLinux distribution with the following shell command:
1 apt-get -y install lighttpd Once installed, you need to edit the Lighttpd configuration file:
1 /etc/lighttpd/lighttpd.conf and, in it, write something like:
1 server.port = 80 2 server.bind = "0.0.0.0" 3 server.event-handler = "freebsd-kqueue" 4 server.modules = ( "mod_rewrite", "mod_fastcgi" ) 5 server.error-handler-404 = "/test.fcgi" 6 server.document-root = "/users/www-data/web2py/" 7 server.errorlog = "/tmp/error.log" 8 fastcgi.server = ( ".fcgi" => 9 ( "localhost" => 10 ( "min-procs" => 1, 11 "socket" => "/tmp/fcgi.sock" 12 ) 13 ) 14 ) Start the web2py fcgihandler before the web-server is started, with: APACHE2 AND MOD PYTHON IN A SHARED HOSTING ENVIRONMENT 295
1 nohup python fcgihandler.py & Then, (re)start the web server with:
1 /etc/init.d/lighttpd restart Notice that FastCGI binds the web2py server to a Unix socket, not to an IP socket:
1 /tmp/fcgi.sock This is where Lighttpd forwards the HTTP requests to and receives responses from. Unix sockets are lighter than Internet sockets, and this is one of the reasons Lighttpd+FastCGI+web2py is fast. As in the case of Apache, it is possible to setup Lighttpd to deal with static files directly, and to force some applications over HTTPS. Refer to the Lighttpd documentation for details. The administrative interface must be disabled when web2py runs on a shared host with FastCGI, or it will be exposed to the other users.
11.8 Apache2 and mod python in a shared hosting environment
There are times, specifically on shared hosts, when one does not have the permission to configure the Apache config files directly. You can still run web2py. Here we show an example of how to set it up using mod python6 • Place contents of web2py into the "htdocs" folder. • In the web2py folder, create a file "web2py modpython.py" file with the following contents:
1 from mod_python import apache 2 import modpythonhandler 3 4 def handler(req): 5 req.subprocess_env['PATH_INFO'] = \ 6 req.subprocess_env['SCRIPT_URL'] 7 return modpythonhandler.handler(req)
• Create/update the file ".htaccess" with the following contents:
1 SetHandler python-program 2 PythonHandler web2py_modpython 3 ##PythonDebug On
6Examples provided by Niktar 296 DEPLOYMENT RECIPES
11.9 Setup Cherokee with FastGGI
Cherokee is a very fast web server and, like web2py, it provides an AJAX- enabled web-based interface for its configuration. Its web interface is written in Python. In addition, there is no restart required for most of the changes. Here are the steps required to setup web2py with Cherokee:
• Download Cherokee [76] • Untar, build, and install:
1 tar -xzf cherokee-0.9.4.tar.gz 2 cd cherokee-0.9.4 3 ./configure --enable-fcgi && make 4 make install
• Start web2py normally at least once to make sure it creates the "ap- plications" folder. • Write a shell script named "startweb2py.sh" with the following code:
1 #!/bin/bash 2 cd /var/web2py 3 python /var/web2py/fcgihandler.py &
and give the script execute privileges and run it. This will start web2py under FastCGI handler. • Start Cherokee and cherokee-admin:
1 sudo nohup cherokee & 2 sudo nohup cherokee-admin &
By default, cherokee-admin only listens at local interface on port 9090. This is not a problem if you have full, physical access on that machine. If this is not the case, you can force it to bind to an IP address and port by using the following options:
1 -b, --bind[=IP] 2 -p, --port=NUM
or do an SSH port-forward (more secure, recommended):
1 ssh -L 9090:localhost:9090 remotehost
• Open "http://localhost:9090" in your browser. If everything is ok, you will get cherokee-admin. • In cherokee-admin web interface, click "info sources". Choose "Local Interpreter". Write in the following code, then click "Add New". SETUP POSTGRESQL 297
1 Nick: web2py 2 Connection: /tmp/fcgi.sock 3 Interpreter: /var/web2py/startweb2py.sh
• Click "Virtual Servers", then click "Default". • Click "Behavior", then, under that, click "default". • Choose "FastCGI" instead of "List and Send" from the list box. • At the bottom, select "web2py" as "Application Server" • Put a check in all the checkboxes (you can leave Allow-x-sendfile). If there is a warning displayed, disable and enable one of the check- boxes. (It will automatically re-submit the application server parameter. Sometimes it doesn’t, which is a bug). • Point your browser to "http://ipaddressofyoursite", and "Welcome to web2py" will appear.
11.10 Setup PostgreSQL
PostgreSQL is a free and open source database which is used in demand- ing production environments, for example, to store the .org domain name database, and has been proven to scale well into hundreds of terabytes of data. It has very fast and solid transaction support, and provides an auto- vacuum feature that frees the administrator from most database maintenance tasks. On an Ubuntu or other Debian-based Linux distribution, it is easy to install PostgreSQL and its Python API with:
1 sudo apt-get -y install postgresql 2 sudo apt-get -y install python-psycopg2 It is wise to run the web server(s) and the database server on different machines. In this case, the machines running the web servers should be connected with a secure internal (physical) network, or should establish SSL tunnels to securely connect with the database server. Start the database server with:
1 sudo /etc/init.d/postgresql restart When restarting the PostgreSQL server, it should notify which port it is running on. Unless you have multiple database servers, it should be 5432. The PostgreSQL configuration file is: 298 DEPLOYMENT RECIPES
1 /etc/postgresql/x.x/main/postgresql.conf
(where x.x is the version number). The PostgreSQL logs are in:
1 /var/log/postgresql/ Once the database server is up and running, create a user and a database so that web2py applications can use it:
1 sudo -u postgres createuser -P -s myuser 2 createdb mydb 3 echo 'The following databases have been created:' 4 psql -l 5 psql mydb The first of the commands will grant superuser-access to the new user, called myuser. It will prompt you for a password. Any web2py application can connect to this database with the command:
1 db = DAL("postgres://myuser:mypassword@localhost:5432/mydb")
where mypassword is the password you entered when prompted, and 5432 is the port where the database server is running. Normally you use one databasefor eachapplication, and multiple instances of the same application connect to the same database. It is also possible for different applications to share the same database. For database backup details, read the PostgreSQL documentation; specifi- cally the commands pg dump and pg restore.
11.11 Security Issues
It is very dangerous to publicly expose the admin application and the ap- padmin controllers unless they run over HTTPS. Moreover, your password and credentials should never be transmitted unencrypted. This is true for web2py and any other web application. In your applications, if they require authentication, you should make the session cookies secure with:
1 session.secure() An easy way to setup a secure production environment on a server is to first stop web2py and then remove all the parameters *.py files from the web2py installation folder. Then start web2py without a password. This will completely disable admin and appadmin. Next, start a second Python instance accessible only from localhost:
1 nohup python web2py -p 8001 -i 127.0.0.1 -a '
and create an SSH tunnel from the local machine (the one from which you wish to access the administrative interface) to the server (the one where web2py is running, example.com), using:
1 ssh -L 8001:127.0.0.1:8001 username@example.com Now you can access the administrative interface locally via the web browser at localhost:8001. This configuration is secure because admin is not reachable when the tunnel is closed (the user is logged out).
This solution is secure on shared hosts if and only if other users do not have read access to the folder that contains web2py; otherwise users may be able to steal session cookies directly from the server.
11.12 Scalability Issues
web2py is designed to be easy to deploy and to setup. This does not mean that it compromises on efficiency or scalability, but it means you may need to tweak it to make it scalable. In this section we assume multiple web2py installations behind a NAT server that provides local load-balancing. In this case, web2py works out-of-the-box if some conditions are met. In particular, all instances of each web2py application must access the same database server and must see the same files. This latter condition can be implemented by making the following folders shared:
1 applications/myapp/sessions 2 applications/myapp/errors 3 applications/myapp/uploads 4 applications/myapp/cache The shared folders must support file locking. Possible solutions are ZFS7, NFS8, or Samba (SMB). It is possible, but not a good idea, to share the entire web2py folder or the entire applications folder, because this would cause a needless increase of network bandwidth usage. We believe the configuration discussed above to be very scalable because it reduces the database load by moving to the shared filesystems those resources
7ZFS was developed by Sun Microsystems and is the preferred choice. 8With NFS you may need to run the nlockmgr daemon to allow file locking. 300 DEPLOYMENT RECIPES
that need to be shared but do not need transactional safety (only one client at a time is supposed to access a session file, cache always needs a global lock, uploads and errors are write once/read many files). Ideally, both the database and the shared storage should have RAID capa- bility. Do not make the mistake of storing the database on the same storage as the shared folders, or you will create a new bottle neck there. On a case-by-casebasis, you may need to perform additional optimizations and we will discuss them below. In particular, we will discuss how to get rid of these shared folders one-by-one, and how to store the associated data in the database instead. While this is possible, it is not necessarily a good solution. Nevertheless, there may be reasons to do so. One such reason is that sometimes we do not have the freedom to set up shared folders.
Sessions in Database It is possible to instruct web2py to store sessions in a database instead of in the sessions folder. This has to be done for each individual web2py application although they may all use the same database to store sessions. Given a database connection
1 db = DAL(...) youcanstorethesessionsinthisdatabase(db)bysimplystating the following, in the same model file that establishes the connection:
1 session.connect(request, response, db) If it does not exist already, web2py creates a table in the database called web2py session appname containing the following fields:
1 Field('locked', 'boolean', default=False), 2 Field('client_ip'), 3 Field('created_datetime', 'datetime', default=now), 4 Field('modified_datetime', 'datetime'), 5 Field('unique_key'), 6 Field('session_data', 'text') "unique key" is a uuid key used to identify the session in the cookie. "ses- sion data" is the cPickled session data. To minimize database access, you should avoid storing sessions when they are not needed with:
1 session.forget() With this tweak the "sessions" folder does not need to be a shared folder because it will no longer be accessed.
Noticethat,ifsessionsaredisabled,youmustnotpassthe session to form.accepts and you cannot use session.flash nor CRUD. SCALABILITY ISSUES 301
Pound, a High Availability Load Balancer If you need multiple web2py processes running on multiple machines, in- stead of storing sessions in the database or in cache, you have the option to use a load balancer with sticky sessions. Pound [78] is an HTTP load balancer and Reverse proxy that provides sticky sessions. By sticky sessions, we mean that once a session cookie has been issued, the load balancer will always route requests from the client associated to the session, to the same server. This allows you to store the session in the local filesystem. To use Pound: First, install Pound, on out Ubuntu test machine:
1 sudo apt-get -y install pound Second editthe configuration file "/etc/pound/pound.cfg"and enable Pound at startup:
1 startup=1 Bind it to a socket (IP, Port):
1 ListenHTTP 123.123.123.123,80 Specify the IP addresses and ports of the machines in the farm running web2py:
1 UrlGroup ".*" 2 BackEnd 192.168.1.1,80,1 3 BackEnd 192.168.1.2,80,1 4 BackEnd 192.168.1.3,80,1 5 Session IP 3600 6 EndGroup The ",1" indicates the relative strength of the machines. The last line will maintain sessions by client IP for 3600 seconds. Third, enable this config file and start Pound:
1 /etc/default/pound
Cleanup Sessions If you choose to keep your sessions in the filesystem, you should be aware that on a production environment they pile up fast. web2py provides a script called:
1 scripts/sessions2trash.py 302 DEPLOYMENT RECIPES
that when run in the background, periodically deletes all sessions that have not been accessed for a certain amount of time. This is the content of the script:
1 SLEEP_MINUTES = 5 2 EXPIRATION_MINUTES = 60 3 import os, time, stat 4 path = os.path.join(request.folder, 'sessions') 5 while 1: 6 now = time.time() 7 for file in os.listdir(path): 8 filename = os.path.join(path, file) 9 t = os.stat(filename)[stat.ST_MTIME] 10 if now - t > EXPIRATION_MINUTES * 60: 11 unlink(filename) 12 time.sleep(SLEEP_MINUTES * 60) You can run the script with the following command:
1 nohup python web2py.py -S yourapp -R scripts/sessions2trash.py & where yourapp is the name of your application.
Upload Files in Database By default, all uploaded files handled by SQLFORMs are safely renamed and stored in the filesystem under the "uploads" folder. It is possible to instruct web2py to store uploaded files in the database instead. Consider the following table:
1 db.define_table('dog', 2 Field('name') 3 Field('image', 'upload'))
where dog.image is of type upload. To make the uploaded image go in the same record as the name of the dog, you must modify the table definition by adding a blob field and link it to the upload field:
1 db.define_table('dog', 2 Field('name') 3 Field('image', 'upload', uploadfield='image_data'), 4 Field('image_data', 'blob')) Here "image data" is just an arbitrary name for the new blob field. Line 3 instructs web2py to safely rename uploaded images as usual, store the new name in the image field, and store the data in the uploadfield called "image data" instead of storing the data on the filesystem. All of this is be done automatically by SQLFORMs and no other code needs to be changed. With this tweak, the "uploads" folder is no longer needed. No Google App Engine files are stored by default in the database without need to define an uploadfield, one is created by default. SCALABILITY ISSUES 303
Collecting Tickets By default, web2py stores tickets (errors) on the local file system. It would not make sense to store tickets directly in the database, because the most common origin of error in a production environment is database failure. Storing tickets is never a bottleneck, because this is ordinarily a rare event, hence, in a production environment with multiple concurrent servers, it is more than adequate to store them in a shared folder. Nevertheless, since only the administrator needs to retrieve tickets, it is also OK to store tickets in a non-shared local "errors" folder and periodically collect them and/or clear them. One possibility is to periodically move all local tickets to a database. For this purpose, web2py provides the following script:
1 scripts/tickets2db.py which contains:
1 import sys 2 import os 3 import time 4 import stat 5 import datetime 6 7 from gluon.utils import md5_hash 8 from gluon.restricted import RestrictedError 9 10 SLEEP_MINUTES = 5 11 DB_URI = 'sqlite://tickets.db' 12 ALLOW_DUPLICATES = True 13 14 path = os.path.join(request.folder, 'errors') 15 16 db = SQLDB(DB_URI) 17 db.define_table('ticket', SQLField('app'), SQLField('name'), 18 SQLField('date_saved', 'datetime'), SQLField('layer') , 19 SQLField('traceback', 'text'), SQLField('code', 'text ')) 20 21 hashes = {} 22 23 while 1: 24 for file in os.listdir(path): 25 filename = os.path.join(path, file) 26 27 if not ALLOW_DUPLICATES: 28 file_data = open(filename, 'r').read() 29 key = md5_hash(file_data) 30 31 if key in hashes: 32 continue 33 34 hashes[key] = 1 304 DEPLOYMENT RECIPES
35 36 error = RestrictedError() 37 error.load(request, request.application, filename) 38 39 modified_time = os.stat(filename)[stat.ST_MTIME] 40 modified_time = datetime.datetime.fromtimestamp(modified_time ) 41 42 db.ticket.insert(app=request.application, 43 date_saved=modified_time, 44 name=file, 45 layer=error.layer, 46 traceback=error.traceback, 47 code=error.code) 48 49 os.unlink(filename) 50 51 db.commit() 52 time.sleep(SLEEP_MINUTES * 60) This script should be edited. Change the DB URI string so that it connects to your database server and run it with the command:
1 nohup python web2py.py -S yourapp -M -R scripts/tickets2db.py & where yourapp is the name of your application. This script runs in the background and every 5 minutes moves all tickets to the database server in a table called "ticket" and removes the local tickets. If ALLOW DUPLICATES is set to False, it will only store tickets that cor- respond to different types of errors. With this tweak, the "errors" folder does not need to be a shared folder any more, since it will only be accessedlocally.
Memcache
We have shown that web2py provides two types of cache: cache.ram and cache.disk. They both work on a distributed environment with multiple concurrent servers, but they do not work as expected. In particular, cache.ram will only cache at the server level; thus it becomes useless. cache.disk will also cache at the server level unless the "cache" folder is a shared folder that supports locking; thus, instead of speeding things up, it becomes a major bottleneck. The solution is not to use them, but to use memcache instead. web2py comes with a memcache API. To use memcache, create a new model file, for example 0 memcache.py, and in this file write (or append) the following code:
1 from gluon.contrib.memcache import MemcacheClient 2 memcache_servers = ['127.0.0.1:11211'] 3 cache.memcache = MemcacheClient(request, memcache_servers) GOOGLE APP ENGINE 305
4 cache.ram = cache.disk = cache.memcache The first line imports memcache. The second line has to be a list of mem- cache sockets (server:port). The third line redefines cache.ram and cache.disk in terms of memcache. Youcouldchooseto redefineonly oneof them to definea totally newcache object pointing to the Memcache object. With this tweak the "cache" folder does not need to be a shared folder any more, since it will no longer be accessed. This code requires having memcache servers running on the local network. You should consult the memcache documentation for information on how to setup those servers.
Sessions in Memcache If you do needsessionsand youdo not want to usea load balancerwith sticky sessions, you have the option to store sessions in memcache:
1 from gluon.contrib.memdb import MEMDB 2 session.connect(request,response,db=MEMDB(cache.memcache))
Removing Applications In a production setting, it may be better not to install the default applications: admin, examples and welcome. Although these applications are quite small, they are not necessary. Removing these applications is as easy as deleting the corresponding fold- ers under the applications folder.
11.13 Google App Engine
It is possible to run web2py code on Google App Engine (GAE) [12], including DAL code, with some limitations. The GAE platform provides several advantages over normal hosting solutions: • Ease of deployment. Google completely abstracts the underlying ar- chitecture. • Scalability. Google will replicate your app as many times as it takes to serve all concurrent requests 306 DEPLOYMENT RECIPES
• BigTable. On GAE, instead of a normal relational database, you store persistent information in BigTable, the datastore Google is famous for.
The limitations are:
• You have no read or write access to the file system.
• No transactions
• You cannot perform complex queries on the datastore, in particular there are no JOIN, OR, LIKE, IN, and DATE/DATETIME operators.
This means that web2py cannot stores sessions, error tickets, cache files and uploaded files on disk; they must be stored somewhere else. Therefore, on GAE, web2py automatically stores all uploaded files in the datastore, whether or not "upload" Field(s) have a uploadfield attribute. You have to be explicit about where to store sessions and tickets: You can store them in the datastore too:
1 db = DAL('gae') 2 session.connect(request,response,db)
Or, you can store them in memcache:
1 from gluon.contrib.gae_memcache import MemcacheClient 2 from gluon.contrib.memdb import MEMDB 3 cache.memcache = MemcacheClient(request) 4 cache.ram = cache.disk = cache.memcache 5 6 db = DAL('gae') 7 session.connect(request,response,MEMDB(cache.memcache))
The absence oftransactionsandtypicalfunctionalities ofrelational databases are what sets GAE apart from other hosting environment. This is the price to pay for high scalability. If you can leave with these limitations, then GAE is an excellent platform. If you cannot, then you should consider a regular hosting platform with a relational database. If a web2py application does not run on GAE, it is because of one of the limitations discussed above. Most issues can be resolved by removing JOINs from web2py queries and denormalizing the database. To upload your app in GA,E we recommend using the Google App Engine Launcher. You can download the software from ref. [12]. Choose [File][Add Existing Application], set the path to the path of the top-level web2py folder, and press the [Run] button in the toolbar. After you have tested that it works locally, you can deploy it on GAE by simply clicking on the [Deploy] button on the toolbar (assuming you have an account). GOOGLE APP ENGINE 307
On Windows and Linux systems, you can also deploy using the shell:
1 cd .. 2 /usr/local/bin/dev_appserver.py web2py When deploying, web2py ignores the admin, examples, and welcome applications since they are not needed. You may want to edit the app.yaml file and ignore other applications as well. On GAE, the web2py tickets/errors are also logged into the GAE admin- istration console where logs can be accessed and searched online.
You can detect whether web2py is running on GAE using the variable 308 DEPLOYMENT RECIPES
1 request.env.web2py_runtime_gae CHAPTER 12
OTHER RECIPES
12.1 Upgrading web2py
In the near future web2py will be able to upgrade itself but this has not yet been implemented at the time of publishing. Upgrading web2py manually is very easy. Simply unzip the latest version of web2py over the old instal- lation. This will upgrade all the libraries but none of the applications, not even the standard applications (admin, examples, welcome), because you may have changed them and web2py does not want to mess with them. The new standard applications will be in the corresponding .w2p files in the web2py root folder. After the upgrade, the new "welcome.w2p" will be used as a scaffolding application. You can upgrade the existing standard applications with the shell command:
1 python web2py.py --upgrade yes
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 309 Copyright © 2009 310 OTHER RECIPES
This will upgrade admin, example, and welcome.
12.2 Fetching a URL
Python includes the urllib library for fetching urls:
1 import urllib 2 page = urllib.urlopen('http://www.web2py.com').read()
This is often fine, but the urllib module does not work on the Google App Engine. Google provides a different API for downloading URL that works on GAE only. In order to make your code portable, web2py includes a fetch function that works on GAE as well as other Python installations:
1 from google.tools import fetch 2 page = fetch('http://www.web2py.com')
12.3 Geocoding
If you need to convertan address(for example: "243 S WabashAve, Chicago, IL, USA") into geographical coordinates (latitude and longitude), web2py provides a function to do so.
1 from gluon.tools import geocode 2 address = '243 S Wabash Ave, Chicago, IL, USA' 3 (latitude, longitude) = geocode(address)
The function geocode requires a network connection and it connect to the Google geocoding service for the geocoding. The function returns (0,0) in case of failure. Notice that the Google geocoding service caps the number of requests and you should check their service agreement. The geocode function is built on top of the fetch function and thus it works on GAE.
12.4 Pagination
This recipe is a useful trick to minimize database accessin case of pagination, e.g., when you need to display a list of rows from a database but you want to distribute the rows over multiple pages. Start by creating a primes application that stores the first 1000 prime numbers in a database. Here is the model db.py:
1 db= DAL('sqlite://primes.db') 2 db.define_table('prime', Field('value','integer')) STREAMING VIRTUAL FILES 311
3 def isprime(p): 4 for i in range(2,p): 5 if p%i==0: return False 6 return True 7 if len(db().select(db.prime.id))==0: 8 p=2 9 for i in range(1000): 10 while not isprime(p): p+=1 11 db.prime.insert(value=p) 12 p+=1
Now create an action list items in the "default.py" controller that reads like this:
1 def list_items(): 2 if len(request.args): page=int(request.args[0]) 3 else: page=0 4 items_per_page=20 5 limitby=(page*items_per_page,(page+1)*items_per_page+1) 6 rows=db().select(db.prime.ALL,limitby=limitby) 7 return dict(rows=rows,page=page,items_per_page=items_per_page) Notice that this code selects one more item than is needed, 20 + 1. The reason is that the extra element tells the view whether there is a next page. Here is the "default/list items.html" view:
1 {{extend 'layout.html'}} 2 3 {{for i,row in enumerate(rows):}} 4 {{if i==items_per_page: break}} 5 {{=row.value}}
6 {{pass}} 7 8 {{if page:}} 9 previous 10 {{pass}} 11 12 {{if len(rows)>items_per_page:}} 13 next 14 {{pass}} In this way we have obtained pagination with one single select per action, and that one select only selects one row more then we need.
12.5 Streaming Virtual Files
It is common for malicious attackers to scan web sites for vulnerabilities. They use security scanners like Nessus to explore the target web sites for scripts that are known to have vulnerabilities. An analysis of web server logs from a scanned machine or directly of the Nessus database reveals that most of the known vulnerabilities are in PHP scripts and ASP scripts. Since 312 OTHER RECIPES
we are running web2py, we do not have those vulnerabilities, but we will still be scanned for them. This annoying, so we like to like to respond to those vulnerability scans and make the attacker understand their time is being wasted. One possibility is to redirect all requests for .php, .asp, and anything suspicious to a dummy action that will respond to the attack by keeping the attacker busy for a large amount of time. Eventually the attacker will give up and will not scan us again. This recipe requires two parts. A dedicated application called jammer with a "default.py" controller as follows:
1 class Jammer(): 2 def read(self,n): return 'x'*n 3 def jam(): return response.stream(Jammer(),40000) When this action is called, it responds with an infinite data stream full of "x"-es. 40000 characters at a time. The second ingredient is a "route.py" file that redirects any request ending in .php, .asp, etc. (both upper case and lower case) to this controller.
1 route_in=( 2 ('.*\.(php|PHP|asp|ASP|jsp|JSP)','jammer/default/jam'), 3 ) The first time you are attacked you may incur a small overhead, but our experience is that the same attacker will not try twice.
12.6 httpserver.log and the log file format
The web2py web server logs all requests to a file called:
1 httpserver.log in the root web2py directory. An alternative filename and location can be specified via web2py command-line options. New entries are appendedto the end of the file each time a requestis made. Each line looks like this:
1 127.0.0.1, 2008-01-12 10:41:20, GET, /admin/default/site, HTTP/1.1, 200, 0.270000 The format is:
1 ip, timestamp, method, path, protocol, status, time_taken Where
• ip is the IP address of the client who made the request SEND AN SMS 313
• timestamp is the date and time of the request in ISO 8601 format, YYYY-MM-DDT HH:MM:SS • method is either GET or POST • path is the path requested by the client • protocol is the HTTP protocol used to send to the client, usually HTTP/1.1 • status is the one of the HTTP status codes [80] • time taken is the amount of time the server took to processthe request, in seconds, not including upload/download time.
In the appliancesrepository[33], you will findan appliancefor log analysis. This logging is disabled by default when using mod wsgi since it would be the same as the Apache log.
12.7 Send an SMS
Sending SMS messages from a web2py application requires a third party service that can relay the messages to the receiver. Usually this is not a free service, but it differs from country to country. In the US, aspsms.com is one of these services. They require signing up for the service and the deposit of an amount of money to cover the cost of the SMS messages that will be sent. They will assign a userkey and a password. Once you have these parameters you need to define a function that can send SMS messages through the service. For this purpose you can define a model file in the application called "0 sms.py" and in this file include the following code:
1 def send_sms(recipient,text,userkey,password,host="xml1.aspsms.com", port=5061,action="/xmlsvr.asp"): 2 import socket, cgi 3 content=""" 4
14 length=len(content) 15 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 16 s.connect((host.port)) 17 s.send("POST %s HTTP/1.0\r\n",action) 18 s.send("Content-Type: text/xml\r\n") 19 s.send("Content-Length: "+str(length)+"\r\n\r\n") 20 s.send(CONTENT) 21 datarecv=s.recv(1024) 22 reply=str(datarecv) 23 s.close() 24 return reply You can call the function from any controller in the application. Notice that the service is ASP-based, but it communicates via XML, so you can call it from a Python/web2py program.
12.8 Twitter API
Here are some quick examples on how to post/get tweets. No third-party libraries are required, since Twitter uses simple RESTful APIs. Here is an example of how to post a tweet:
1 def post_tweet(username,password,message): 2 import urllib, urlib2, base64 3 import gluon.contrib.simplejson as sj 4 args= urllib.urlencode([('status',message)]) 5 headers={} 6 headers['Authorization'] = 'Basic '+base64.b64encode(username+':' +password) 7 request = urllib2.Request('http://twitter.com/statuses/update. json', args, headers) 8 return sj.loads(urllib2.urlopen(req).read()) Here is an example of how to receive tweets:
1 def get_tweets(): 2 user='web2py' 3 import urllib 4 import gluon.contrib.simplejson as sj 5 page = urllib.urlopen('http://twitter.com/%s?format=json' % user) .read() 6 tweets=XML(sj.loads(page)['#timeline']) 7 return dict(tweets=tweets) For more complex operations, refer to the Twitter API documentation.
12.9 Jython
web2py normally runs on CPython (the Python interpreter coded in C), but it can also run on Jython (the Python interpreter coded in Java). This allows web2py to run in a Java infrastructure. JYTHON 315
Even though web2py runs with Jython out of the box, there is some trickery involved in setting up Jython and in setting up zxJDBC (the Jython database adaptor). Here are the instructions:
• Download the file "jython installer-2.5.0.jar" (or 2.5.x) from Jython.org • Install it:
1 java -jar jython_installer-2.5.0.jar
• Download and install "zxJDBC.jar" from http://sourceforge.net/projects/zxjdbc/
• Download and install the file "sqlitejdbc-v056.jar" from http://www.zentus.com/sqlitejdbc/
• Add zxJDBC and sqlitejdbc to the java CLASSPATH • Start web2py with Jython
1 /path/to/jython web2py.py
You will be able to use DAL(’sqlite://...’) and DAL(’postgres://...’) only.
References
1. http://www.web2py.com 2. http://www.python.org 3. http://en.wikipedia.org/wiki/SQL 4. http://www.sqlite.org/ 5. http://www.postgresql.org/ 6. http://www.mysql.com/ 7. http://www.microsoft.com/sqlserver 8. http://www.firebirdsql.org/ 9. http://www.oracle.com/database/index.html 10. http://www-01.ibm.com/software/data/db2/ 11. http://www-01.ibm.com/software/data/informix/ 12. http://code.google.com/appengine/ 13. http://en.wikipedia.org/wiki/HTML 14. http://www.w3.org/TR/REC-html40/ 15. http://www.php.net/ 16. http://www.cherrypy.org/browser/trunk/cherrypy/wsgiserver/ init .py 17. http://en.wikipedia.org/wiki/Web Server Gateway Interface
WEB2PY: Enterprise Web Framework / 2nd Ed.. By Massimo Di Pierro 317 Copyright © 2009 318 REFERENCES
18. http://www.python.org/dev/peps/pep-0333/ 19. http://www.owasp.org 20. http://en.wikipedia.org/wiki/Secure Sockets Layer 21. http://www.cherrypy.org 22. http://www.cdolivet.net/editarea/ 23. http://nicedit.com/ 24. http://pypi.python.org/pypi/simplejson 25. http://pyrtf.sourceforge.net/ 26. http://www.dalkescientific.com/Python/PyRSS2Gen.html 27. http://www.feedparser.org/ 28. http://code.google.com/p/python-markdown2/ 29. http://www.tummy.com/Community/software/python-memcached/ 30. http://www.fsf.org/licensing/licenses/info/GPLv2.html 31. http://jquery.com/ 32. https://www.web2py.com/cas 33. http://www.web2py.com/appliances 34. http://www.web2py.com/AlterEgo 35. http://www.python.org/dev/peps/pep-0008/ 36. Guido van Rossum, and Fred L. Drake, An Introduction to Python (version 2.5), Network Theory Ltd, 164 pages (November 2006) 37. Mark Lutz, Learning Python, O’Reilly & Associates, 701 pages (October 2007) 38. http://www.python.org/doc/ 39. http://en.wikipedia.org/wiki/Cascading Style Sheets 40. http://www.w3.org/Style/CSS/ 41. http://www.w3schools.com/css/ 42. http://en.wikipedia.org/wiki/JavaScript 43. David Flanagan, JavaScript: The Definitive Guide JavaScript: The Definitive Guide, O’Reilly Media, Inc.; 5 edition (August 17, 2006) 44. http://www.xmlrpc.com/ 45. http://en.wikipedia.org/wiki/Hypertext Transfer Protocol 46. http://www.w3.org/Protocols/rfc2616/rfc2616.html 47. http://en.wikipedia.org/wiki/XML 48. http://www.w3.org/XML/ 49. http://en.wikipedia.org/wiki/XHTML 50. http://www.w3.org/TR/xhtml1/ 51. http://www.w3schools.com/xhtml/ 52. http://www.web2py.com/layouts REFERENCES 319
53. http://sourceforge.net/projects/zxjdbc/ 54. http://pypi.python.org/pypi/psycopg2 55. http://sourceforge.net/projects/mysql-python 56. http://python.net/crew/atuining/cx Oracle/ 57. http://pyodbc.sourceforge.net/ 58. http://kinterbasdb.sourceforge.net/ 59. http://informixdb.sourceforge.net/ 60. http://www.web2py.com/sqldesigner 61. http://www.faqs.org/rfcs/rfc2616.html 62. http://www.faqs.org/rfcs/rfc2396.html 63. http://tools.ietf.org/html/rfc3490 64. http://tools.ietf.org/html/rfc3492 65. http://www.recaptcha.net 66. http://www.reportlab.org 67. http://en.wikipedia.org/wiki/AJAX 68. Karl Swedberg and Jonathan Chaffer, Learning jQuery, Packt Publishing 69. http://ui.jquery.com/ 70. http://en.wikipedia.org/wiki/Common Gateway Interface 71. http://www.apache.org/ 72. http://httpd.apache.org/docs/2.0/mod/mod proxy.html 73. http://sial.org/howto/openssl/self-signed 74. http://code.google.com/p/modwsgi/ 75. http://www.lighttpd.net/ 76. http://www.cherokee-project.com/download/’’ 77. http://www.fastcgi.com/ 78. http://www.apsis.ch/pound/ 79. http://pyamf.org/ 80. http://en.wikipedia.org/wiki/List of HTTP status codes
Index
autodelete, 198 A B
A, 134 B, 134 about, 84, 104 belongs, 176 accepts, 54, 66, 182 blob, 157 Access Control, 223 BODY, 134 access restriction, 11 C Active Directory, 232 admin, 44, 81, 298 cache, 104, 111 admin.py, 84 controller, 112 Adobe Flash, 257 disk, 111 ajax, 71, 263 memcache, 304 ALL, 163 ram, 111 and, 165 select, 177 Apache, 281 view, 113 appadmin, 59, 91 CAPTCHA, 228 appliances, 45 CAS, 241 ASCII, 24 CENTER, 134 ASP, 4 CGI, 281 asynchronous submission, 277 checkbox, 137 as list, 247 Cherokee, 296 Auth, 223 class, 33 authentication, 241 CLEANUP, 209 Authentication, 261 CODE, 135
321 322 INDEX command line options, 94 effects, 268 commit, 159 elif, 31, 130 confirmation, 272 else, 31, 130–131 connection pooling, 152 EM, 136 connection strings, 151 emails content-disposition, 195 template, 146 controllers, 104 encode, 24 cookies, 117 errors, 87, 104 cooperation, 126 escape, 128 count, 166 eval, 36 cPickle, 40 examples, 44 cron, 121 except, 31, 131 cross site request forgery, 10 Exception, 31 cross site scripting, 9 exec, 36 CRUD, 214 executesql, 160 create, 214 exec environment, 124 delete, 214 export, 170 read, 214 Expression select, 214 DAL, 151 tables, 214 extent, 143 update, 214 CRYPT, 210 F cryptographic storage, 11 csv, 170 FastCGI, 294, 296 CSV, 250 favicon, 119 custom validator, 211 fcgihandler, 294 fetch, 310 D Field constructor, 154 Field, 153, 162 DAC, 223 DAL, 150 DAL, 105, 149, 153 fields, 157, 191 DALStorage, 150, 162 FIELDSET, 136 DAL|shortcuts, 177 file.read, 34 Database Abstraction Layer, 149 file.seek, 35 database drivers, 150 file.write, 34 databases, 104 finally, 31, 131 date, 39, 175 FireBird, 153 datetime, 39, 175 for, 28, 129 day, 175 form self submission, 53 DB2, 153 form, 51 def, 29, 131 FORM, 54, 136 default, 153 form, 182 define table, 150, 153 formname, 182 deletable, 273 delete, 166 G delete label, 192 dict, 26 GAE dir, 23 login, 232 distinct, 165 geocode, 310 distributed transactions, 161 GET, 97 DIV, 136 Gmail, 232 Document Object Model (DOM), 133 Google App Engine, 305 Domino, 232 grouping, 169 drop, 160
E H EDIT, 85 H1, 136 INDEX 323
HEAD, 137 JSONRPC, 253 help, 23 JSP, 4 helpers, 105, 132 Jython, 314 hidden, 136 hour, 175 K HTML, 137 html, 173 keepvalues, 186 HTTP, 104, 115 KPAX, 81 httpserver.log, 312 L I LABEL, 138 id label, 192 labels, 191 if, 31, 130 lambda, 35 IFRAME, 138 languages, 104 IF MODIFIED SINCE, 97 Layout Builder, 147 import, 37, 124, 170 layout, 52 improper error handling, 10 layout.html, 143 include, 143 LDAP, 230, 232 index, 45 left outer join, 168 information leakage, 10 LEGEND, 138 Informix, 153 length, 153 inheritance, 179 LI, 138 init, 118 license, 13, 84, 104 injection flaws, 9 Lighttpd, 294 inner join, 168 like, 175 INPUT, 54, 137 limitby, 165 insecure object reference, 10 list, 25 insert, 158 Lotus Notes, 232 internationalization, 104, 116 lower, 175 IS ALPHANUMERIC, 203 IS DATE, 203 M IS DATETIME, 203 IS EMAIL, 57, 204 MAC, 223 IS EXPR, 204 malicious file execution, 10 IS FLOAT IN RANGE, 204 many-to-many relation, 173 IS IMAGE, 207 markdown, 76 IS INT IN RANGE, 204 MENU, 142 IS IN DB, 57, 210 menu IS IN SET, 204 response, 144 IS IPV4, 209 Mercurial, 91 IS LENGTH, 205 META, 138 IS LIST OF, 205 migrate, 153 IS LOWER, 205, 209 migrations, 154 IS MATCH, 205 minutes, 175 IS NOT EMPTY, 54, 57, 206 Model-View-Controller, 5 IS NOT IN DB, 210 models, 104 IS NULL OR, 209 modules, 104 IS STRONG, 207 mod proxy, 281 IS TIME, 206 mod python, 281 IS UPLOAD FILENAME, 208 mod wsgi, 281 IS UPPER, 209 month, 175 IS URL, 206 MSSQL, 153 MySQL, 153 J N join, 168 JSON, 246, 259 nested select, 176 324 INDEX not, 165 args, 66, 97 notnull, 153 controller, 97 cookies, 105 O env, 108 function, 97 OBJECT, 138 get vars, 97 OL, 138 post vars, 97 ON, 138 url, 97 ondelete, 153 vars, 51, 97 one to many, 167 required, 153 onvalidation, 186 requires, 54, 153 OpenLDAP, 232 response, 104, 107 OPTION, 139 author, 107 or, 165 body, 107 Oracle, 153 cookies, 107 orderby, 164 description, 107 os, 38 download, 107 os.path.join, 38 flash, 66, 107 os.unlink, 38 headers, 107 outer join, 168 keywords, 107 menu, 107 P postprocessing, 107 P, 139 render, 107 page layout, 143 status, 107 pagination, 310 stream, 66, 107 PARTIAL CONTENT, 97 subtitle, 107 password, 93 title, 107 PDF, 260 view, 107 PHP, 4 write, 107 PIL, 228 response.menu, 144 POST, 97 response.write, 128 PostgresSQL, 153 return, 29, 131 pound, 301 robots, 119 PRE, 139 Role-Based Access Control, 223 private, 104 rollback, 159 PyAMF, 257 routes in, 118 Pyjamas, 253 routes on error, 120 PyRTF, 260 routes out, 118 Python, 21 Rows, 162–163, 168 DAL, 150 Q RPC, 251 rss, 71, 79 Query, 162 RSS, 248 DAL, 150 RTF, 260
R S radio, 137 sanitize, 77, 134 random, 37 scaffolding, 44 RBAC, 223 scalability, 299 reCAPTCHA, 228 SCRIPT, 139 redirect, 53, 104, 115 seconds, 175 referencing, 167 secure communications, 11 removing application, 305 security, 9, 298 ReportLab, 260 select, 64 request, 3, 104–105 SELECT, 139 application, 97 select, 162 INDEX 325
selected, 139 TLS, 232 session, 50, 104, 110 TR, 140–141 session.connect, 110 truncate, 159 session.forget, 110 try, 31, 131 session.secure, 110 TT, 141 Set, 162 tuple, 26 DAL, 150 type, 24, 153 shell, 22 showid, 192 U simplejson, 259 site, 81 UL, 141 SMS, 313 Unicode, 24 SMTP, 232 unique, 153 SPAN, 139 update, 166 SQL designer, 158 update record, 166 SQL upgrades, 309 generate, 169 upload, 56 sql.log, 153 uploadfield, 153 SQLFORM, 66 uploads, 104 SQLite, 153 upper, 175 SQLRows, 168 url mapping, 96 SQLTABLE, 164 url rewrite, 118 static files, 96 URL, 53, 113 static, 104 UTF8, 24 Storage, 105 DAL, 150 V str, 24 streaming virtual file, 311 validators, 105, 202 STYLE, 140 views, 104, 127 submit button, 192 sum, 176 W sys, 38 sys.path, 38 Web Services, 245 welcome, 44 T while, 29, 130 wiki, 71 T, 104, 116 Windows service, 293 TABLE, 140 WSGI, 281 Table, 157, 162 tables, 157 X TAG, 142 TBODY, 140 XHTML, 137 TD, 140 XML, 133 template language, 127 xml, 173 tests, 104 XML, 246 TEXTAREA, 141 xmlrpc, 71, 80 TFOOT, 141 XMLRPC, 253 TH, 141 THEAD, 141 Y time, 39, 175 TITLE, 141 year, 175